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FOREWORD 


The  Computer-Based  Instructional  Systems  team  of  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  performs 
research  and  levelopment  in  the  area  of  educational  technology  with 
applicability  to  military  education  and  training.  Of  special  interest  is 
research  in  the  atea  of  large  scale  computer-based  instructional  systems. 
The  development  and  implementation  of  these  systems  is  seen  as  a  solution 
to  current  Army  training  problems  such  as  the  management  of  self-paced 
instruction. 

This  Research  Report  reviews  the  proposed  functions  of  one  such 
system  —  the  Automated  Instructional  Management  System  (AIMS)  — 
and  provides  a  formative  evaluation  plan  to  assess  its  management, 
training,  and  cost  effectiveness.  The  research  effort  is  responsive  to 
the  requirements  of  RDT&E  Project  2Q263744A795,  Manpower  and  Educational 
Systems  Technical  Area  of  the  FY  80  ARI  Work  Program. 


A  FORMATIVE  EVALUATION  PLAN  FOR  THE  AUTOMATED  INSTRUCTIONAL  MANAGEMENT 
SYSTEM  (AIMS) 


BRIEF 


Requirement: 

To  develop  a  plan  to  evaluate  the  training  ar.d  cost-effectiveness  of  the 
Automated  Instructional  Management  System  (AIMS) .  The  AIMS  is  a  computer-based 
training  and  resource  development  system  being  developed  by  the  Training  and 
Doctrine  Command  (TRADOC)  for  possible  Installation  at  its  schools  and  training 
centers. 


Procedure: 

The  Functional  Description  of  the  proposed  AIMS  was  examined  to  identify  the 
hardware  and  software  packages  intended  for  development.  Stufflebeam's  CIPP 
Model  (Context,  Input,  Process,  Product)  served  as  the  theoretical  framework  for 
the  evaluation.  Evaluation  Questions  were  formulated  to  guide  the  examination  of 
the  AIMS  hardware/software,  training  management,  courseware,  training  effective¬ 
ness,  cost,  and  lsqilementation/organizatlonal  factors.  A  Milestone  Chart  was 
prepared  to  indicate  deadlines  for  the  evaluation  activities  from  the  development 
of  data  collection  Instruments  through  preparation  of  the  final  report.  A 
description  of  the  staff  required  to  Implement  the  evaluation  plan  was  prepared. 
Draft  data  collection  instruments  were  devised  with  the  guidance  that  revisions 
would  be  required  to  tailor  the  instruments  to  the  specific  software  packages 
operational  at  each  AIMS  site. 


Utilization  of  Findings: 

A  reduced  version  of  this  plan  has  been  prepared  to  evaluate  the  AIMS  test 
site  at  the  US  Army  Field  Artillery  School,  Ft.  Sill,  OK.  The  formative 
evaluation  at  Ft.  Sill  will  provide  data  for  system  revisions  which  may  be 
implemented  prior  to  the  AIMS  installation  at  the  remaining  TRADOC  sites.  The 
present  plan  provides  for  analyses  pre-  and  post-  AIMS  installation  with  the 
intention  of  providing  information  for  the  design  of  the  next  generation  of 
computer-based  instructional  systeos. 
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A  FORMATIVE  EVALUATION  PLAN  FOR  THE 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


BACKGROUND 


The  Army  has  made  a  substantial  commitment  to  self- 
paced  instruction  which  enables  the  student  to  progress  at 
his/her  own  rate.  However,  if  the  student  must  wait  for 
equipment,  assignments,  tests,  instructor  aid,  and  even 
other  students,  the  benefits  of  progressing  through  a  course 
eit  an  individual  pace  are  lost.  For  this  reason,  computers 
have  had  an  increasing  role  in  the  modern  training  institu¬ 
tion.  Computer-managed  instructional  systems  provide  the 
necessary  data  processing  associated  with  self-paced  in¬ 
struction  so  that  timely  information  can  be  made  available 
to  instructors  and  students. 

In  1975,  the  headquarters  of  the  U.S.  Army  Training  and 
Doctrine  Command  (TRADOC)  directed  that  self-paced  training 
be  implemented  throughout  its  schools.  At  one  TRADOC 
school,  the  U.S.  Army  Field  Artillery  School  (USAFAS) ,  Ft. 
Sill,  Oklahoma,  the  impact  of  this  mandate  affected  eight 
courses  training  6  -  8,000  students  annually,  a  complex  and 
burdensome  manual  process  for  administering  these  courses 
emerged.  Instructors  who  were  trained  to  develop,  review 
and  deliver  instruction,  were  required  to  spend  most  of 
their  time  performing  clerical  tasks  such  as  marking  -check- 
sheets  to  indicate  student  progress. 

As  self-paced  courses  proliferated,  an  urgent  need  was 
perceived  at  the  USAFAS  and  throughout  TRADOC  for  a  system¬ 
atic  method  for  managing  and  administering  the  new  self- 
paced  training.  In  response  to  this  need,  a  feasibility 
study  was  conducted  to  determine  the  most  appropriate  com¬ 
puter-managed  instruction  (CMI)  system  needed  to  support 
USAFAS  self-paced  training.  An  analysis  was  made  of  the 
Versatile  Training  System  (VTS)  developed  by  the  Naval 
Weapons  Center  at  China  Lake,  California;  the  Memphis  CMI 
System  developed  by  the  Navy;  and  the  Computerized  Training 
System  (CTS)  developed  by  the  Army  under  the  ABACUS  project. 
The  systems  were  rated  on  five  criteria: 


Note.  The  authors  wish  to  acknowledge  the  technical  and 

support  assistance  of  Ms.  Judith  Paris,  International 
Public  Policy  Research  Corporation. 
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(1)  Training  support  requirements  which  include  man¬ 
agement  information,  scheduling  assistance,  com¬ 
puter-managed  instruction,  aids  to  authoring/de¬ 
velopers,  and  modest  CAI  potential? 

(2)  Optimal  hardware  configuration; 

(3)  Maintenance; 

(4)  Continued  personnel  support?  and 

(5)  Evolution  potential. 


The  VTS  received  a  higher  average  weighted  rating  than  the 
other  two  systems  because  of  its  match  with  the  USAFAS ' 
priorities  for  training  support  requirements  (Bunderson, 
1977) . 

The  VTS  was  developed  by  the  Navy  in  1972  under  a 
competitively  procured  contract  which  provided  total  support 
for  software,  hardware,  maintenance,  and  documentation.  The 
VTS  consists  of  a  central  processing  unit  (CPU)  and  associ¬ 
ated  peripherals,  optical  readers,  and  terminals.  The  types 
and  quantities  of  hardware  and  capabilities  of  software  are 
determined  for  each  site's  unique  requirements.  VTS  soft¬ 
ware  supports  the  development  of  training  programs  for 
specific  ratings  (MOS) ,  indicates  resource  configuration  and 
scheduling  information,  and  produces  a  variety  of  reports. 
System  support  software  enables  up  to  64  simultaneous  users 
to  interactively  enter,  store,  and  retrieve  a  large  amount 
of  data. 

The  Navy  has  made  available  to  TRADOC  a  VTS  equipment 
configuration,  software  support  for  necessary  modifications, 
a  site  manager,  maintenance  support,  and  contractor  support. 
The  TRADOC  version  is  termed  the  Automated  Instructional 
Management  System  (AIMS).  The  AIMS  is  a  training  management 
system  designed  to  provide  automated  data  collection,  pro¬ 
cessing,  and  retrieval  for  personnel  files,  diagnostic 
testing,  training  schedules,  graduation  prediction,  and  a 
variety  of  course  information. 

The  AIMS  was  installed  at  the  USAFAS  in  October  1978  as 
the  test  site  for  all  TRADOC  schools.  The  test  system 
consists  of  a  Digital  Equipment  Corporation  (DEC)  PDP-11/70 
computer,  associated  peripherals,  additional  mass  storage  in 
the  form  of  a  moving  head  disk,  a  magnetic  tape  drive,  a 
high  speed  printer,  terminals,  and  optical  readers.  AIMS 
applications  software  consists  of  modified  VTS  software  and 
programs  newly  developed  according  to  the  USAFAS'  require¬ 
ments. 


2 


The  following  software  packages  are  proposed  for  modi¬ 
fication  and  development  on  the  AIMS: 

•  System  Support 

•  Personnel 

•  Test  and  Evaluation 

•  Computer-Managed  Instruction  (CMI) 

•  Resource  Configuration  and  Scheduling 

•  Report  Generation 


The  System  Support  Software  consists  of  the  DEC'S 
Resource  Sharing  Timesharing  System/Extended  (RSTS/E)  soft¬ 
ware.  The  RSTS/E  allows  files  to  be  created,  updated,  and 
deleted  from  the  user's  terminal.  A  Privacy  Act  Compliance 
software  package  has  been  developed  in  conjunction  with  the 
RSTS/E  for  the  Pt.  Sill  test  site  so  that  no  unauthorized 
software  can  be  installed  or  modifications  made  to  existing 
AIMS  software. 


Personnel  Software  is  being  modified/developed  so  that 
data  such  as  name ,  rank ,  SSN,  MOS,  training  qualifications, 
and  past  assignments  can  be  maintained  for  each  student, 
staff,  and  faculty  member.  The  personnel  files  will  be  used 
to  create  a  personnel  data  base;  maintain  assignment  and 
training  histories;  assist  in-processing  and  out-processing 
functions;  provide  class  registration;  and  prepare  class 
rosters,  completion  certificates,  diplomas,  and  training 
reports . 


The  Test  and  Evaluation  Software  is  intended  for  the 
creation  and  maintenance  of  test  items  and  tests.  The 
software  will  support  the  creation  of  a  master  test  item 
bank,  generate  tests  from  the  master  file,  administer  and 
score  on-line  tests,  produce  printed  tests  for  off-line  use, 
score  off-line  tests  by  a  mark-sense  reader,  and  store 
prescriptions  keyed  to  each  test  item  to  be  delivered  during 
on-line  testing  or  by  hard  copy  for  off-line  testing. 


Computer-Managed  Instruction  (CMI)  Software  is  planned 
to  manage  single  ana  multi-path  progression  through  a  self- 
paced  course.  The  progress  of  each  student  will  be  recorded 
and  reported  to  the  instructor  so  that  modifications  to  a 
student  s  specific  course  components  ox  path  can  be  made. 

The  software  will  also  enable  the  prediction  of  module, 

lesson,  and  course  completion  times  based  on  historical 
data  • 


.■L...i:rT-!yjrCeiCon5iqUr^tion  and  Scheduling  Software  is  being 
modified/developed  to  control  the  availability  of  "training 
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resources.  The  software  will  provide  class  schedules  and 
enable  the  assignment  of  training  materials  (texts,  test 
equipment,  tools,  training  aids)  on  a  time  reserved  basis. 

Reports  Generation  Software  will  be  modified/developed 
to  produce  the  reports  and  forms  to  interface  with  the  other 
AIMS  software  packages.  Instructional  and  administrative 
personnel  will  be  aided  by  the  rosters,  tests,  grade  sheets, 
diplomas,  and  reports  produced  by  the  Reports  Generation 
software. 

The  previously  mentioned  hardware  and  preceeding  soft¬ 
ware  packages  are  planned  for  implementation  as  a  two-phase 
effort.  During  Phase  I,  selected  software  packages  will  be 
tested  at  the  U.S.  Army  Field  Artillery  School,  Ft.  Sill. 
Additional  software  options  will  be  exercised  during  Phase 
II.  The  Management  Information  Systems  Office  (MISO)  of 
TRADOC's  Army  Training  Support  Center  (ATSC)  requested  that 
the  U.S.  Army  Research  Institute  (ARI)  prepare  a  plan  to 
evaluate  the  AIMS.  This  evaluation  plan  covers  all  software 
options  proposed  in  the  Systems  Consultants,  Inc.,  Proposal 
No.  73-78-024,  Development  of  an  Automated  Instructional  Man¬ 
agement  System  for  the  U.S.  Army  (Revised  13  November  1978) . 
The  plan  is  responsive  to  TRADOC’s  request  and  reflects  the 
objective  of  the  ARI  Work  Unit,  "Computer-Managed  Instruction" 
(Project  A795,  Thrust  4,  Task  B,  Work  Unit  2,  FY80  Work  Pro¬ 
gram)  . 

The  following  sections  of  this  Research  Report  describe  ... 
the  proposed  evaluation,  its  conceptual  framework,  purpose, 
procedures,  staffing,  and  reporting  requirements. 


¥  ' 
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EVALUATION  MODEL 


The  theoretical  framework  guiding  the  proposed  evalua¬ 
tion  of  AIMS  is  the  CIPP  (CONTEXT,  INPUT,  PROCESS,  PRODUCT) 
evaluation  model  (StuFFTebeam,  et  al.,  1971). 


The  first  component  -  CONTEXT  -  refers  to  the  initial 
assessment  of  need  and  underlying  rationale  for  the  program. 
This  assessment  occurs  prior  to  program  initiation  and 
results  in  statements  of  system  deficiencies  and  needs. 

With  regard  to  the  AIMS,  TRADOC 1 s  commitment  to  self-pacing 
underscored  the  requirement  for  an  efficient  training  man¬ 
agement  system.  The  need  at  tho  USAFAS  was  documented  by 
the  procurement  request  for  an  automated  system  to  overcome 
the  problems  in  their  manual  procedures  for  managing  self- 
paced  training.  Thus,  the  documentation  for  the  AIMS  pro¬ 
curement  at  the  USAFAS  should  provide  all  the  information 
necessary  to  describe  the  Context  which  led  to  the  AIMS 
program,  and  within  which  its  evaluation  will  take  place. 


The  second  component  of  the  CIPP  evaluation  model  - 
INPUT  -  refers  to  the  examination  and  comparison  of  alterna¬ 
tive  system  designs  in  meeting  program  needs.  This  activity 
occurs  early  in  a  system's  conceptual  stage  and  results  in 
the  selection  of  one  or  more  approaches  or  designs  to  meet 
the  needs  and  objectives  determined  by  the  Context  evaluation. 
The  Input  evaluation  has  been  performed  and  documented  in 
the  study  comparing  the  Computerized  Training  System, 

Memphis  CMI,  and  the  Versatile  Training  System  (Bunderson, 
1977).  In  addition.  Systems  Consultants,  Inc.  (SCI)  has 
documented  the  training  management  requirements  of  ail 
TRADOC  service  schools  and  Army  Training  Centers. 


The  third  CIPP  component  -  PROCESS  -  refers  to  an 
evaluation  of  how  adequately  a  program's  intended  implemen¬ 
tation  plans  and  procedures  are  being  carried  out.  This 
information  is  used  during  a  program's  early  stages  to  help 
correct  any  deficiencies  and  improve  the  implementation 
process  and  conduct  of  the  program.  With  respect  to  the 
AIMS,  a  Process  evaluation  should  be  continuously  performed 
during  the  entire  installation  period  at  the  USAFAS  test 
site.  This  should  include  monitoring  and  documentation  of 
implementation  problems  and  their  solutions.  A  formal 
assessment  of  the  AIMS  components  by  government  acceptance 
tests  of  hardware/software  and  querying  AIMS  users  should 
also  be  included  in  the  evaluation  of  the  AIMS  process. 


The  fourth  CIPP  evaluation  model  component  -  PRODUCT  - 
the  outcomes  of  the  program  and  exami nei~them  in 
relation  to  satisfying  the  needs  identified  previously.  The 
information  gathered  in  this  evaluation,  as  in  Process 
evaluation,  is  used  for  program  modifications  and  improve¬ 
ments.  Process  and  Product  evaluation  interact  and  should 
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occur  simultaneously.  Whereas  Product  evaluation  determines 
the  extent  to  which  objectives  have  been  or  are  being 
attained.  Process  evaluation  describes  the  procedures  that 
were  actually  implemented,  and  this  can  provide  an  explana¬ 
tion  for  whether  or  not  the  objectives  were  achieved. 
Similarly,  both  types  of  evaluation  provide  feedback  for 
changing  a  program's  procedures.  With  respect  to  AIMS,  the 
Product  evaluation  should  be  performed  once  the  hardware/ 
software  system  has  become  stabilized  and  operational. 
Measures  of  the  AIMS'  outcomes  should  be  obtained  from  on¬ 
line  records  as  well  as  instruments  to  be  administered  off¬ 
line. 


It  is  clear  from  the  above  discussion  of  the  applica¬ 
tion  of  the  CIPP  model  to  the  AIMS  evaluation,  that  data  is 
available  and  has  been  used  for  decision-making  regarding 
the  Context  and  Input  of  the  AIMS.  The  next  section  will 
focus  on  the  purpose  of  the  AIMS  evaluation  regarding  the 
collection  of  the  needed  Process  and  Product  data. 


EVALUATION  PURPOSE/ISSUES 


The  purpose  of  this  plan  is  to:  Evaluate  the  effec¬ 
tiveness  of  the  AIMS  in  meeting  TRADOC 1 s  training  management 
needs.  The  appropriate  focus  for  the  Process  and  Product 
evaluation  of  the  AIMS  is  formative.  Operational  and  train¬ 
ing  effectiveness  data  will  be  collected  at  the  Ft.  Sill 
test  site  so  that  revisions  can  be  made  prior  to  AIMS  dis¬ 
tribution  Army-wide. 

The  goal  of  the  AIMS  is  to  provide  overall  management 
support  for  TRADOC  training  programs.  AIMS  is  intended  to 
provide  managers,  instructors,  administrators,  and  support 
personnel  with  an  interactive  computer  system  with  which 
they  can  obtain: 

•  Student  progress  reports 

•  Training  schedules 

•  Predictions  of  student  graduation  dates 

•  Prescriptions  for  remediation 

•  Test  administration 

•  Test  scoring 

•  Monitoring  of  student  progress  during  training 

A  selection  of  AIMS  software  packages,  identified  in 
the  previous  section,  will  be  operational  at  the  USAFAS. 

The  two  USAFAS  courses  which  will  be  used  to  test  the  AIMS 
are  the  Artillery  Survey  Specialist  Course  (MOS  82C10)  and 
the  Field  Artillery  Target  Acquisition  Specialist  Course 
(MOS  17C10) . 

The  approach  taken  in  the  proposed  evaluation  plan  in¬ 
cludes  the  following  steps: 


Step 

1. 

Ensure  that  all  parties  concerned 
know  about,  understand,  and  agree 
to  the  proposed  evaluation. 

Step 

2. 

Validate  the  USAFAS  training  man¬ 
agement  needs  to  be  met  by  the  AIMS. 

Step 

3. 

Examine  match  between  needs  and  the 
AIMS'  functions  and  characteristics 
using  data  collection  instruments 
specific  to  the  test  site. 
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Step  4.  Assess  the  AIMS  operations  in  the 
two  USAFAS  courses. 

Step  b.  Provide  input  for  modifications  to 
the  AIMS  at  USAFAS. 

Step  6.  Reassess  the  AIMS  operations  in  the 
two  USAFAS  courses. 

Step  7.  Provide  input  for  modifications  to 
the  AIMS  to  be  installed  at  other 
TRADOC  schools. 

The  evaluation  will  focus  on  the  following  components: 

I .  Hardware/Software 

II.  Training  Management 

III .  Courseware 

IV.  Training  Effectiveness 

V.  Cost 

VI.  Implementation/Organizational  Factors 

A  list  of  evaluation  questions  has  been  formulated  to 
guide  the  examination  of  each  component.  These  questions 
were  suggested  by  Orlansky  and  String  (1979),  Seidel,  et 
al.,  (1V78)  and  an  analysis  of  the  AIMS'  objectives  derived 
from  the  USAFAS  needs  identified  by  Systems  Consultants, 
Inc.  (SCI) „  These  evaluation  questions  appear  in  the  next 
section  under  their  AIMS  component  headings. 


EVALUATION  QUESTIONS 


I .  Hardware/Software 

1. 1  Is  the  AIMS  hardware/software  configuration  ade¬ 
quate  for  the  Army's  training  and  resource  manage¬ 
ment  needs  at  Ft.  Sill  (and  other  installations)? 

1.2  What  back-up  capabilities  are  available  during 
system  down-time? 


II.  Training  Management 

11. 1  Does  data  exist  on  each  system  participant  and  is 
access  to  this  data  limited  to  appropriate  users? 

11. 2  Does  a  course  catalog  exist? 

11. 3  Is  a  list  of  training  tasks  maintained  for  each 
course? 

11. 4  Are  tests  administered,  scored,  and  recorded 
according  to  instructors'  specifications? 

11. 5  Are  sufficient  resources  available  to  handle  the 
student  load? 

11. 6  Does  a  catalog  exist  describing  all  training 
materials? 

11. 7  How  are  instructors,  students,  and  materials 
assigned  to  courses? 

11. 8  How  useful  are  the  AIMS-generated  reports  for 
instructors  and  training  managers? 

11. 9  How  accurate  is  the  graduation  prediction  function? 

11. 10  Have  in/out  processing  time  and  inconvenience  been 
reduced? 

11. 11  Has  student  registration  time  been  reduced? 

11.12  Are  all  training  reports  (class  rosters,  etc.)  pro¬ 
duced  in  a  timely  manner? 

11.13  Are  all  required  training  records  and  paperwork 
produced  by  AIMS? 

11.14  Has  the  system  been  effective  in  monitoring  stu¬ 
dents? 
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III.  Courseware 


111.1  What  procedure  is  used  by  the  instructor  to  define 
the  instructional  environment  and  strategy  appro¬ 
priate  for  each  student? 

111. 2  Are  tests  constructed  according  to  instructors' 
specifications? 

111. 3  How  does  the  instructor  determine  the  hierarchy  of 
CMI  lesson  topics? 

111. 4  What  feedback  is  available  for  revising  instruc¬ 
tional  materials  and  tests? 

111. 5  Are  the  instructors  able  to  develop  courses 
according  to  the  Instructional  Systems  Development 
Model? 

111. 6  Hew  much  time  is  required  to  prepare  training  and 
test  materials  contained  in  CMI  courses? 

111. 7  What  special  problems,  if  any,  were  encountered 
when  inputting  materials  on-line? 


IV.  Training  Effectiveness 

IV. 1  What  percentage  of  the  students  score  above  the 
cutoff  for  80%  of  the  tests? 

IV. 2  What  percentage  of  the  students  fail  to  graduate 
the  CMI  courses? 

IV. 3  What  are  student  attitudes  towards  the  AIMS 
courses? 

IV. 4  Whac  are  instructor  attitudes  toward  the  AIMS? 

IV, 5  What  unanticipated  by-products  can  be  attributed 
to  the  AIMS? 

IV. 6  How  well  do  CMI  course  graduates  perform  on  the 
job? 

IV. 7  What  is  the  average  length  of  time  needed  to  com¬ 
plete  the  AIMS  CMI  courses? 


V.  Costs 


V.l  What  costs  have  been  incurred  to  develop,  deliver, 
staff,  and  maintain  the  AIMS? 


V.2  What  are  the  estimated  costs  to  manually  admin¬ 
ister  USAFAS  self-paced  courses? 

V.3  What  costs  have  been  avoided  by  use  of  the  AIMS 
at  USAFAS? 


VI.  Implementation/Organi zational  Factors 

VI. 1  What  was  the  status  of  the  AIMS  within  USAFAS? 

VI . 2  Was  there  agreement  or  understanding  on  the  pur¬ 
poses  and  goals  of  the  AIMS  prior  to  implemen¬ 
tation? 

VI. 3  Were  there  clear  lines  of  responsibility/author¬ 
ity  within  the  AIMS  program  at  USAFAS? 

VI. 4  Did  instructors  and  other  system  users  receive 
familiarization  or  other  training  on  the  AIMS 
as  a  "tool"  for  their  benefit? 

VI. 5  Was  the  AIMS  field  test  representative  of  condi¬ 
tions  that  would  exist  under  normal  operational 
conditions? 

VI. 6  Were  there  frequent  meetings  of  user  and  AIMS  pro¬ 
ject  personnel  to  monitor  the  field  test  and  en¬ 
hance  mutual  understanding? 

VI. 7  Were  key  personnel  assignments  to  the  AIMS  CMI 
courses  sufficiently  free  of  turbulence? 

VI. 8  What  special  qualifications  should  the  AIMS  in¬ 
structional  support  staff  have? 
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EVALUATION  ACTIVITIES 


Evaluative  data  on  the  AIMS  will  be  collected  by  two 
approaches.  In  the  first  approach  the  software  subsystems 
of  the  AIMS  will  be  tested  with  data  inputs  (during  govern¬ 
ment-conducted  acceptance  tests  and  actual  AIMS  operations). 
This  will  provide  some  answers  to  the  evaluation  questions 
concerning  the  hardware/software  and  training  management  of 
the  AIMS  (Components  1  and  2) .  The  second  approach  will  be 
to  obtain  the  USAFAS  records  and  to  query  or  interview 
system  users  for  information  on  courseware,  training  effec¬ 
tiveness,  costs,  and  implementation/organizational  factors 
(Components  3,  4,  5,  and  6).  This  examination  of  records 
and  interviews  of  system  users  will  also  provide  information 
for  Components  1  and  2. 

The  specific  activities  proposed  in  this  evaluation 
plan  are  described  below.  A  schedule  of  these  activities 
and  associated  milestones  is  shown  in  Figure  1. 

Activity  1.  Obtain  Contract  Support 

There  is  a  need  in  any  evaluation  (formative  or  summa- 
tive)  for  all  concerned  individuals  to  understand: 

•  Who  and  what  is  being  evaluated 

•  How  the  assessment  is  to  be  made 

•  What  criteria  are  to  be  used  to  judge  the 
program 

•  Who  is  to  do  the  data  collection,  analyses, 
and  interpretation 

•  What  decisions  are  to  be  made  as  a  result  of 
the  study 

•  How  much  time,  cost,  and  resources  are  re¬ 
quired  to  implement  the  evaluation 

Attaining  this  level  of  understanding  by  all  concerned 
parties  requires  a  set  of  explicit  activities  that  should  be 
an  integral  part  of  the  management  of  this  evaluation 
effort.  As  the  purpose  of  this  evaluation  has  been  estab¬ 
lished  as  formative  —  the  findings  are  to  be  used  for  the 
improvement  of  subsequent  AIMS  implementations  —  then  this 
purpose  must  be  clearly  communicated  to  all  concerned  indi¬ 
viduals.  This  is  especially  true  when  the  evaluators  are 
seen  as  "outsiders".  In  the  proposed  evaluation,  ARI  and 
its  evaluation  contractor  could  be  perceived  in  this  manner 
by  the  user  (USAFAS)  and/or  the  system  developer  (SCI) . 
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Therefore,  it  is  proposed  that  a  set  of  formal  and 
informal  meetings/briefings  be  arranged  to  establish  working 
relationships  between  all  participants.  The  purpose  of 
these  meetings  should  be  two- fold:  (1)  to  communicate  the 
purpose,  procedures,  and  details  of  the  evaluation  plan,  and 
(2)  to  involve  the  AIMS  developer  and  users  in  setting 
evaluation  priorities,  contributing  to  data  collection  plant} 
and  activities,  and  determining  how  the  evaluative  informa¬ 
tion  would  be  used  to  improve  thu  AIMS.  Attendees  at  these 
meetings  should  represent  all  interested  or  "concerned" 
parties  to  the  AIMS  implementation.  These  include:  TRADOC 
(ATSC-MISO) ;  USAFAS  personnel  (representatives  from  manage¬ 
ment,  instructors,  academic  departments,  departments  of 
training  development  and  evaluation,  etc.);  SCI  project  and 
site  managers;  and  ARI  and  its  evaluation  contractor. 
Cooperation  between  all  concerned  is  essential  to  the  suc¬ 
cess  of  AIMS  and  its  planned  evaluation.  One  possible  means 
for  ensuring  involvement  of  the  system  developer  (SCI)  in 
the  formal  evaluation  is  to  contract  for  the  development  of 
software  that  permits  on-line  evaluative  data  collection/ 
analysis  on  the  AIMS. 

Once  a  degree  of  understanding  and  cooperation  has  been 
achieved,  and  agreements  are  obtained  regarding  evaluation 
priorities,  the  specific  evaluation  procedures  need  to  be 
discussed.  Arrangements  should  be  made  for  scheduling  and 
supporting  the  data  collection,  analyzing  the  data,  and 
reporting  the  findings  for  use  in  modifying  the  AIMS. 

The  outcome  of  this  activity  is  the  formal  approval  of 
the  evaluation  plan.  This  plan  should  be  a  formal  document 
supplemented  by  Memoranda  of  Understanding  that  detail  in 
specific  terms  the  results  of  the  above-described  meetings 
to  obtain  interagency  cooperation  and  coordination  in  terms 
of  contract  support. 

Activity  2.  Development/Modification  of  Data  Collection 
Instruments 


Interview  schedules  and  survey  instruments  will  be 
developed  and/or  modified  to  collect  data  of  interest. 
Instrument  development/modification  will  correspond  with 
the  AIMS  evaluation  components.  Hardware/Software  (Com¬ 
ponent  1)  will  be  assessed  by  a  series  of  on-line  tests  of 
the  AIMS  subsystems.  Prior  to  the  government's  acceptance 
of  the  AIMS,  tests  will  be  conducted  to  determine  whether 
the  system  meets  the  contract's  specifications.  The  data 
collected  should  indicate  the  operational  status  of  the 
hardware  and  software  In  addition  to  these  tests,  the  AIMS 
subsystems  will  be  examined  in  operation.  The  hardware/ 
software  should  not  be  tested  until  the  AIMS  has  been  in 
place  and  the  staff  is  trained  and  experienced  in  its  use. 


The  operational  tests  will  also  provide  data  on  Training 
Management  (Component  2).  System  users  should  be  polled  for 
information  on  the  operation  of  the  subsystems.  Survey 
instruments  from  the  Computerized  Training  System  (CTS) 
Evaluation  (Seidel,  et  al. ,  1978)  have  been  modified  to 
collect  information  on  the  Resource  Configuration  and  Sche¬ 
duling  Subsystem  (Attachment  1)  and  the  Reports  Generation 
Subsystem  (Attachment  2) . 

Survey  instruments  are  also  available  for  Courseware 
(Component  3).  These  instruments  when  modified  can  provide 
the  means  for  collecting  data  on  the  Computer-Managed  In¬ 
struction  (CMI)  Subsystem  and  the  Test  and  Evaluation  Sub¬ 
system.  The  CMI  Sul  system  permits  the  creation  and  execu¬ 
tion  of  self-paced  training.  Two  survey  instruments  and  an 
interview  schedule  which  have  been  developed  for  the  CTS  are 
appropriate  for  examining  the  development  and  revision  of 
the  AIMS  course  materials  (Attachments  3,  4,  &  5).  The  crea¬ 
tion,  maintenance,  and  administration  of  a  test  item  bank 
are  also  portions  of  the  Courseware  effort.  Attachment  6 
provides  initial  guidance  for  the  design  of  an  instrument  to 
collect  data  on  the  usefulness  and  quality  of  the  AIMS- 
constructed  tests. 

The  fourth  component  of  the  AIMS  evaluation  plan  is 
Training  Effectiveness.  According  to  Seidel  and  Wagner 
(1977) ,  training  effectiveness  can  be  measured  in  terms  of 
the  graduate's  ability  to  perform  the  tasks  selected  for 
training  and  the  graduate's  ability  to  perform  on  the  job. 

The  former  measurement  is  made  at  the  school  while  the 
latter  must  be  made  in  the  field.  Orlansky  and  String 
(1979) ,  in  an  analysis  of  the  training  and  cost  effective¬ 
ness  of  military  computer-based  instruction  projects,  indi¬ 
cated  that  data  on  post-graduation  performance  has  not  been 
collected.  An  examination  of  the  job  performance  feedback 
system  at  USAFAS  is  needed  to  determine  what  instruments  and 
procedures  have  to  be  developed  to  make  these  measurements. 

In  the  evaluation  of  the  Advanced  Instructional  System 
(AIS) ,  Dallman,  et  al.,  (1979)  obtained  supervisor  ratings 
of  AIS  graduates'  job  performance  one  to  three  months  after 
graduation.  Also,  graduates  rated  the  appropriateness  and 
usefulness  of  their  training.  A  similar  assessment  proce¬ 
dure  could  be  adopted  for  the  AIMS. 

Seidel  and  Wagner  (1977)  suggest  the  following  types  of 
training  effectiveness  measures:  absence  rates,  accuracy 
scores,  achievement  measures,  attitude  scales,  attrition 
rates,  and  training  time.  The  AIMS  is  designed  to  record 
within-course  training  effectiveness  data.  The  CMI  Subsystem 
provides  a  timekeeping  function.  Achievement  and  accuracy 
data  are  recorded  by  the  Test  and  Evaluation  Subsystem.  It 
is  anticipated  that  the  Reports  Generation  Subsystem  could 
produce  data  in  the  desired  format.  Attitude  questionnaires 
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will  be  needed  for  all  types  of  AIMS  users.  Categories  of 
system  users  include  students,  instructors,  training  spe¬ 
cialists,  educational  specialists,  instructional  program¬ 
mers,  battery  clerks,  registrars,  and  the  Commander.  Stu¬ 
dent  attitude  questionnaires  have  been  modified  for  the  AIMS 
evaluation  (Attachments  7  &  8) . 

It  is  possible  that  all  data  collection  instruments 
could  be  administered  at  the  AIMS  terminals  and  tabulated  by 
the  Test  and  Evaluation  Subsystem.  Some  additional  software 
development  may  be  needed  for  this  capability. 

Cost  effectiveness  data  (Component  5)  will  require  the 
development  of  AIMS-specif ic  data  collection  instruments. 

Some  of  the  cost  data  Orlansky  and  String  (1979)  recommend 
for  collection  include:  program  design,  delivery,  manage¬ 
ment,  personnel,  and  equipment.  The  Cost-Effectiveness 
Specification  for  Computer-Based  Training  Systems  (Seidel 
and  Wagner,  1977)  can  be  used  as  a  framework  provided  that 
the  appropriate  cost  data  are  available  and  can  be  collected 
at  USAFAS.  A  list  of  cost  categories  derived  from  the 
Seidel  and  Wagner  (1977)  document  appears  in  Attachment  9. 

When  performing  an  evaluation  in  an  operational  environ¬ 
ment,  it  is  important  to  completely  document  that  environ¬ 
ment,  attending  to  those  factors  which  probably  influence 
results  (Da liman,  et  al.,  1979).  It  is  clear  that  such 
factors  as  instructor  bias,  unstable  system  components, 
unclear  lines  of  responsibility,  and  unrealistic  student 
loads  can  negatively  affect  outcomes  of  computer-based 
instructional  systems.  Structured  interviews  need  to  be 
developed  for  use  with  the  USAFAS  management  and  instruc¬ 
tional  personnel,  and  with  the  AIMS  project  personnel  to  ob¬ 
tain  information  on  implementation  and  organizational  fac¬ 
tors  (Component  6) .  Work  sample  observations  (similar  to 
those  employed  by  Dallman,  et  al.,  1979,  Attachment  10)  can 
be  used  to  identify  the  AIMS'  instructional  manpower  require¬ 
ments.  Implementation  and  organizational  factors  influenc¬ 
ing  the  success  of  a  training  management  system  can  also  be 
revealed  in  instructor  and  staff  attitude  surveys  (Attach¬ 
ments  11  and  12) . 

A  list  of  the  data  collection  instruments/procedures 
discussed  in  this  section  is  presented  in  Table  1  on  the 
following  page. 

Activity  3 .  Data  Collection 

The  questionnaires,  student  records,  interviews,  and 
other  instruments  described  in  the  previous  section  will  be 
initially  administered  at  USAFAS  during  a  four-month  period 
(as  shown  in  Figure  1) .  This  period  is  to  coincide  with  the 
implementation  of  the  Artillery  Survey  Specialist  Course 
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TABLE  1 

EVALUATION  COMPONENT/INFORMATION  SOURCES 


AIMS  Evaluation  Component 
I.  Hardware/Software 


II.  Training  Management 

III .  Courseware 


IV.  Training  Effectiveness 


V.  coats 


VI.  Implementation/ 

Organisational factors 


Information  Sources 

Acceptance  Testa 
Syatema  Logs  (Mainten¬ 
ance) 

Attachment  2  (Section  IV) 
Attachment  3  (Questions 
8-30) 

Interviews 

Attachment  1  -  Resource 
Configuration  and 
Scheduling  Survey 
Attachment  2  -  Reports 
Generation  Survey 

Attachment  3  -  Course 
Materials  Development 
Survey 

Attachment  4  -  Revised 
Training  Materials 
Survey 

Attachment  S  -  Course 
Development  Interview 
Attachment  6  -  Test  and 
Evaluation  Survey 
Interviews 
Observations 

Student  Records 
Achievement  scores  and 
other  measurea  as  noted 
in  Cost-Effectiveness 
Specification,  Vol.  3 
(op.  67-93)  Seidel  and 
Wagner,  1977. 

Attachment  7  -  Student 
Attitude  Questionnaire 
Attachment  8  -  CMI 

Attitude  Questionnaire 
Field  Feedback  Reports 
Interviews 

Cost  Data 
Time  Logs 

Contractor  Reports 
Cost-Effectiveness  Spe¬ 
cification  (Vols.  1  t 
Ill,  pp.  5-66)  Seidel 
and  Wagner,  1977 
Attachment  9  -  Cost 
Categories 

Observations  (Work 
Sample) 

Observations  (Attachment 
10  -  Instructor  Task 
List) 

Attachment  11  -  Instruc¬ 
tor  Attitude  Survey 
Attachment  12  -  Staff  and 
Faculty  Attitude  Survey 
Interviaws 


(MOS  82C10)  and  the  Field  Artillery  Target  Acquisition 
Specialist  Course  (MOS  17C10).  The  specific  schedule  for 
questionnaire/interview  administration  will  be  established 
soon  after  formal  approval  of  the  evaluation  plan. 

As  many  instruments  as  possible  should  be  administered 
and/or  tabulated  on-line,  It  must  be  recognized  that  the 
school's  goal  is  to  train  its  students  in  a  timely  manner. 
Therefore,  a  back-up  manual  training  management  system  will 
need  to  remain  operational  until  the  AIMS  has  been  debugged 
sufficiently  and  stabilized. 

Persons  engaged  in  the  evaluation  effort  will  need  to 
work  closely  with  USAFAS  and  SCI  personnel  during  this 
period.  Data  gathered  at  this  time  are  to  be  used  for 
immediate  revisions  or  modifications  of  the  AIMS.  SCI 
and/or  USAFAS  personnel  must  be  made  aware  of  problems  as 
soon  as  they  occur.  Such  problems  and  their  solutions 
should  then  be  documented. 

Activity  4.  Data  Reduction/Anaiysis/Reporting 

Evaluation  data  gathered  on-line  regarding  hardware/ 
software,  training  management,  and  courseware  (Components  1, 
2,  and  3)  will  need  to  be  reported  to  those  who  must  use  the 
information  to  make  changes  in  the  AIMS.  Such  repo  -ts 
should  be  produced  automatically  by  the  AIMS  but  may  need  to 
be  summarized  for  presentation  to  the  users. 

Activity  5.  Interim  Report 

This  report  will  document  the  formative  evaluation 
activities  and  findings  that  are  recorded  during  the  initial 
application  of  the  AIMS  to  the  two  USAFAS  courses.  This 
report  will  also  include: 

•  Description  of  the  initial  AIMS  operations 

•  Comparison  of  USAFAS  training  management 
needs  and  AIMS  functions 

•  Discussion  of  lessons  learned 

•  List  of  recommendations  for  modifying  the 
AIMS  to  facilitate  its  implementation  in 
other  USAFAS  courses 

Activity  6.  Implement  Revisions 

Modifications  and  additions  to  the  AIMS  will  be  made 
based  on  the  suggestions  of  the  Interim  Report.  Research  on 
maximizing  the  effectiveness  of  computer-managed  training 
will  be  conducted  at  this  time. 
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Activity  7 ♦  Data  Collection 


The  same  procedures  described  in  Activity  #3  above  will 
be  re-initiated  for  an  additional  four-month  period.  The 
findings  and  recommendations  described  in  the  Interim  Report 
will  have  been  transmitted  to  the  AIMS  developers  and  users. 
It  is  expected  that  modifications  to  AIMS  based  on  these 
recommendations  will  have  been  made,  so  the  focus  of  data 
collection  in  this  second  period  will  be  on  the  effective¬ 
ness  of  these  revisions.  It  is  possible  that  long-term  data 
such  as  job  performance  feedback  may  be  obtained  during  this 
period.,  Cost  comparison  data  within  the  USAFAS  can  also  be 
gathered. 

Activity  8.  Data  Reduction/Analysis 

This  activity  is  essentially  the  same  as  described  in 
Activity  #4  above.  The  focus  will  be  on  lessons  learned 
that  can  be  used  as  guidance  for  the  AIMS  installations  at 
other  TRADOC  schools. 

Activity  9.  Final  Report 

The  formative  evaluation  will  be  completed  with  the 
submission  and  distribution  of  a  final  report  that  includes 
the  following: 

•  Description  of  AIMS  field  test  at  USAFAS 

•  Description  of  evaluation  activities 

•  Findings/conclusions  for  each  AIMS  evaluation 
component 

•  Cost  analysis  and  projections  (if  feasible) 

•  Guidance  for  subsequent  AIMS  installations  at 
TRADOC  schools 


Based  on  data  gathered  and  lessons  learned  during  this 
field  test,  a  document  will  be  prepared  to  guide  the  next 
generation  of  Army  computer-based  training  systems  in  the 
management  of  instruction.  Whereas  the  immediate  results 
of  this  formative  evaluation  will  aid  the  AIMS  implementa¬ 
tion  at  the  USAFAS,  the  long-term  use  of  the  data  is  to  pro¬ 
vide  an  empirical  and  analytical  basis  for  computer-based 
approaches  to  training  delivery. 


COMP ARAT IVE  ANALYSIS 


The  present  evaluation  plan  does  not  include  a  compar¬ 
ative  study  between  computer-managed  and  manually-managed 
instruction.  The  division  of  AIMS  users  into  experimental 
and  control  groups  is  not  recommended  for  its  initial  im¬ 
plementation  at  USAPAS  for  three  reasons : 

(1)  The  system  is  in  a  formative,  relatively 
unstable  condition  and  thus  not  represen¬ 
tative  of  other  sites, 

(2)  As  AIMS  is  phased  in,  the  manual  mode  is 
simultaneously  being  phased  out,  and 

(3)  The  most  value  to  be  derived  from  an  eval¬ 
uation  at  this  stage  is  to  use  the  inform¬ 
ation  to  improve  the  AIMS. 

However,  some  initial  comparative  data  may  be  collected 
at  the  USAFAS  if  the  two  courses  are  still  conducted  in  the 
manual  mode.  This  data  should  be  collected  during  the 
latter  part  of  the  proposed  study  period.  By  that  time, 
improvements  will  have  made  and  more  accurate  cost  and 
training  effectiveness  estimates  can  be  determined.  These 
data  should  also  be  collected  at  the  second  and  third  AIMS 
installations  so  that  a  more  complete  cost  and  training 
effectiveness  analysis  can  be  made.  The  Cost-Effectiveness 
Specification  for  Computer-Based  Training  Systems  previously 
discussed  (Seidel  and  Wagner,  1977)  could  serve  as  the 
framework  for  these  analyses. 

The  following  section  summarizes  the  staffing  required 
to  implement  the  AIMS  evaluation  plan  presented  in  this 
document.  The  qualifications  of  each  needed  staff  member  is 
indicated  with  a  summary  of  job  tasks  and  estimate  of  person 
months  required  to  accomplish  the  job  tasks. 


DESCRIPTION  OF  STAFF  REQUIRED  TO 
IMPLEMENT  AIMS  EVALUATION  PLAN 


A  -  All  activities  up  to  and  including  Interim  Report 
B  -  All  activities  up  to  and  including  Final  Report 


Estimated  Person  Months 


A  B 


1  Senior  Research  Scientist  - 

Evaluation  Specialist  6 

Experienced  in  evaluation  plan¬ 
ning,  design,  and  methods  -  par¬ 
ticularly  with  regard  to  computer- 
based  instruction. 

Tasks:  AIMS  evaluation  planning 
and  design,  inter-agency  coordina¬ 
tion,  management  of  evaluation  ac¬ 
tivities,  reporting. 

2  Research  Scientists  8 

Experienced  in  evaluation  techni¬ 
ques,  instrument  development,  in¬ 
terviewing,  cost-effectiveness 
analysis,  reporting. 

Tasks:  AIMS  evaluation  instrument 
development,  data  collection,  con¬ 
duct  structural  interviews,  data 
analysis/interpretation ,  reporting . 

1  Site  Manager  -  Research  Scientist  7 

Experienced  in  evaluation  methodol¬ 
ogy,  liaison  with  military  school 
personnel. 

Tasks;  Oversee  on-site  data  collec¬ 
tion  activities,  liaison  between 
USAFAS,  SCI,  and  ARI,  reporting. 

1  Research  Associate/Assistant  7 

Experienced  in  instrument  develop¬ 
ment  data  collection,  reduction, 
and  analysis. 

Tasks:  Revise,  test,  and  admin- 
ister  data  collection  instruments, 
summarize  data,  support  site  man¬ 
ager. 
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12 


12 


9 
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Estimated  Person  Months 


2  Programmers 


Experienced  in  AIMS  software  sub¬ 
system  development. 

Tasks:  Prepare  programs  that  pro 

vide data  collection  instruments 
on-line,  analyze  data,  and  tabu¬ 
late  findings  into  properly  for¬ 
matted  reports. 


2  Clerk  Typists 


Experienced  in  typing  data  col¬ 
lection  instruments,  use  of  job 
entry  data  terminals. 

Tasks:  Type  reports,  enter  data 
for  analysis,  type  and  reproduce 
interim  and  final  reports. 


4 


4 


This  staffing  plan  assumes  that  all  evaluation  activi¬ 
ties  are  performed  in  a  seventeen-month  period;  student  and 
school  records  are  made  available  upon  request  by  USAFAS 
personnel;  school  and  AIMS  development  personnel  (USAFAS, 
SCI,  etc.)  are  available  for  on-site  interviews  as  scheduled 
and  programs  can  b=  written  to  provide  automated  data  col¬ 
lection,  analysis,  and  reporting  on  the  AIMS. 
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RESOURCE  CONFIGURATION  AND  SCHEDULING  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 

DATE 


The  purpose  of  this  survey  is  to  collect  information 
on  the  allocation  of  resources  within  your  area.  The  re¬ 
sults  will  be  consolidated  with  other  surveys  to  determine 
the  effectiveness  of  the  Automated  Instructional  Manage¬ 
ment  System  (AIMS) . 

As  you  read  the  statements  and  questions,  make  a  men¬ 
tal  comparison  of  your  training  area  prior  to  and  after  the 
implementation  of  the  Automated  Instructional  Management 
System. 

Please  place  a  check  (VO  mark  by  your  position  below. 


Position 
Instructor 
Section  Chief 
Course  Chief 
Division  Chief 
Education  Specialist 
Training  Specialist 
Records  Clerk 
Other 


t 


turn-mt  nuBi 
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RESOURCE  CONFIGURATION  AND  SCHEDULING  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 


YES 


NO 


COMMENTS 


1. 


The  computer  system  has 
been  able  to  accurately 
account  for  the  location 
of  each  student  within 
the  course. 


2.  The  computer  system  has 
maintained  an  accurate 
accounting  of  student 
position  vacancies. 


3.  When  the  student  completes 
a  task,  the  system  routes 
him/her  to  the  next  task 
without  delay. 


4.  Students  are  routed  through 
the  course  according  to 
the  predetermined  (normal) 
flow. 


5.  Student  routing  has  been 
accomplished  with  a  mini¬ 
mum  of  errors. 


RESOURCE  CONFIGURATION  AND  SCHEDULING  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 


UJ 

UJ 

cc 

o 

< 

>■ 

_l 
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o 

DC 
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UJ 
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COMMEN  rS 


6 .  When  student  positions 
in  the  next  sequential 
task  are  filled,  the 
student  is  routed  to  an 
alternate  task,  for 
which  he/she  has  the  neces¬ 
sary  prerequisites. 


7.  When  the  student  completes 
a  task  and  all  student 
positions  in  the  succeed¬ 
ing  tasks  are  filled,  the 
instructor  is  alerted. 


8.  Routing  students  to  the 
correct  student  positions 
has  required  close  moni¬ 
toring  by  the  instructor/ 
supervisor. 


9.  The  instructor  is  alerted 
when  the  student  has  com¬ 
pleted  all  the  required 
tasks. 
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RESOURCE  CONFIGURATION  AND  SCHEDULING  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


10.  Prior  to  the  AIMS,  did  accounting  for  student  time  in 
the  course  place  an  unnecessary  burden  on  the  class¬ 
room  instructor?  Yes _  No _ 

Explain. 


11.  Does  the  usefulness  drived  from  using  the  computer  for 
accounting  for  students '  time  outweigh  the  workload  in 
its  collection?  Yes  No 

Explain. 
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RESOURCE  CONFIGURATION  AND  SCHEDULING  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


12.  What  is  your  frank  opinion  of  the  capability  of  the 
computer  system  to  route  and  account  for  students 
within  your  course?  (Please  indicate  advantages  and 
d i sad vantages ) 


Advantages 


Disadvantages 


Attachment 


M,poRTS  GENEWTION^RvEVsTEH  (MMS) 

automated  instructional  manage™. 


REPORTS  GENERATION  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


DATE 


The  purpose  of  this  survey  is  to  collect  information 
on  the  reports  generated  by  the  AIMS  to  help  maintain  stu¬ 
dent  records  and  provide  information  for  managing  self- 
paced  training.  This  information  will  be  analyzed  with 
the  idea  of  making  record  keeping  simpler  and  reporting 
more  useful. 

Please  indicate  your  position  below  by  a  check  mark 
(V')  .  If  more  than  one  position  applies,  make  additional 
check  marks. 


Course  Chief 
Course  NCOIC 


Course  Training 
Specialist 

Course  Section 
Chief 

Course  Instructor 


The  following  is  a  glossary  of  terms  which  will  assist 

you  in  answering  the  questions: 

•  On-line  Testing  -  Using  the  computer  to  administer, 
score,  and  record. 

•  Diagnostic  Tests  (quizzes)  -  An  informal  test. 

•  Regularly  -  At  regular  times  or  intervals. 

•  Occasionally  -  Now  and  then. 

•  Seldom  -  On  only  a  few  occasions. 

•  On-line  -  Interaction  of  operating  a  terminal  with  the 
computer. 

•  Downtime  -  An  interval  of  time  when  the  computer  is  not 
productive . 


Dept.  Operations  Chief 

Dept.  Education  Special¬ 
ist 

Dept.  Course  Materials 
Analyst 

Dept.  Records  Clerk 

Division  Chief 
Division  NCCIC 


REPORTS  GENERATION  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  ~0 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 


LU  f= 

CC  5 

C3  LLJ 

<  z 


COMMENTS 


SECTION  1, 


Weekly  AIMS 
Reports . 


The  AIMS  has  reduced  the 
time  the  classroom  in¬ 
structor  spends  on  stu¬ 
dent  records. 


Weekly  AIMS  printouts 
enable  rapid  analysis  of 
student  accomplishment 
in  relation  to  his/her 
peers. 


The  ability  to  display- 
student  records  on  the 
terminal 

a.  provides  effective 
feedback  on  student 
effectiveness 


saves  time  in 
assessment  of  student 
needs . 


The  AIMS  has  had  no  im¬ 
pact  on  student  record 
keeping  at  the  division/ 
department  level. 


,  h».  ,  =  .’/ 


REPORTS  GENERATION  SURVEY 


SECTION  II.  AIMS-Generated  Reports 
For  each  type  of  report  generated  by  AIMS 
(Title  of  Report) 


Are  you  familiar  with  this  report? 
(If  No,  turn  page  to  next  report) . 

Is  this  report  available  to  you? 


Yes 


No 


a. 

b. 

c. 


Regularly  available 
Occasionally  available 
“Seldom  or  never  available 


How  frequently  do  you  use  this  report? 


a. 

b. 

c. 


Regularly 
“Occasionally 
"Seldom  or  never 


How  accurate  is  this  report? 


a. 

b. 

c. 

d. 


Generally  accurate 
Minor  inaccuracies 


Numerous  errors 

"Cannot  judge  accuracy  of  report 


What  is  your  opinion  about  the  format  of  this  report? 


a. 

b. 

c. 

d. 


_Excellent  format 
"Format  is  satisfactory 
"Format  requires  revision 
"No  opinion 


The  information  contained  in  this  report  is 


a.  _ Essential 

b.  Non-essential 


7.  The  content  of  this  report 


in  the  content 
in  the  content 


a. 

b. 

c. 


_Should  not  be  changed 
"Needs  minor  revisions 
"Needs  major  revisions 
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REPORTS  GENERATION  SURVEY 


AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


8.  How  useful  do  you  find  this  report? 


a. 

Serves  intended  purpose 

b. 

Of  marginal  use 

c. 

Serves  no  useful  purpose 

COMMENTS : 


9.  T’ .e  objective  of  the  AIMS _ _ _ Report 

is  to  provide  the  course  manager  with  timely  informa- 
tion  concerning  student  progress  and  achievement. 

a.  How  well  do  you  think  the  AIMS _ 

Report  has  met  this  objective? 


I 

b.  Has  the  expanded  AIMS _ ■ _ Report 

provided  you  with  a  better  management  tool? 
Explain. 


c.  Is  the  revised  AIMS _ Report 

received  in  time  to  assist  the  course/division/ 
department  in  completing  the  student  records? 
Explain. 


d.  Has  AIMS  improved  the  timeliness  of  the  student 
graduation  prediction?  Explain. 


e.  Should  any  additional  items  be  included  or  deleted 

from  the  AIMS _ Report?  If  yes, 

explain. 
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REPORTS  GENERATION  SURVEY 


AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


Does  the  AIMS  provide  your  course (s)  with  the  neces¬ 
sary  operational  reports?  If  not,  please  list  and 
describe  any  additional  reports  you  think  are  neces¬ 
sary. 


11.  What  AIMS  reports  are  not  necessary?  List  and  ex¬ 
plain. 


SECTION  III.  Student  Record. 

The  student  record  file  was  created  to  provide  real  time 
access  to  individual  student  records,  both  on-line  and 
printouts,  as  required  by  the  primary  instructor  or  course 
managers. 

1.  Does  the  student  record  contain  the  necessary  infor¬ 
mation  to  enable  you  to: 

a.  Analyze  a  student's  progress?  Explain. 


b.  Prescribe  remedial  training?  Explain. 


c.  Take  action  to  separate  the  student  from  the 
course?  Explain. 


Have  you  experienced  any  problems  in  using  the  ter¬ 
minal  to  call  up  a  student's  record?  If  yes,  explain. 
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REPORTS  GENERATION  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


3.  When  you  call  for  a  student’s  record  on-line,  is  there 
any  appreciable  time  lag  in  obtaining  the  display?  If 
yes,  explain. 


4.  Has  the  printout  of  the  student's  record  been  available 
on  a  timely  basis?  If  no,  explain. 


5.  Does  the  student  record  contain  the  necessary  data  to 
support  the  counselling  and  guid  ince  program  in  your 
course?  If  no,  explain. 


6.  Do  you  usa  the  student  record  printout  to  support 
fa<  .ty  board  actions?  Please  explain. 


7.  What  additional  information  do  you  need  in  the  student 
record? 


SECT'-",  uv.  Vbter  Downtime  and  Updating  Reports. 

(Answer  the  following  questions  with  a  "yes”  or  "no"  and 
explain  your  answer.) 

1.  Does  comput  lowntime  interrupt  the  normal  flow  of 
AIMS  repor 


REPORTS  GENERATION  SURVEY 


AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 

Did  the  computer  downtime  result  in  any  of  the  follow¬ 
ing: 

a.  Irretrievable  loss  of  student  data? 

b.  Delay  in  student  graduation? 

c.  Distortion  of  students'  data  on  his/her  progress? 

d.  Hand  processing  of  student  data? 

e.  Overtime  (extra)  work  for  administrative  personnel? 

f.  Delay  in  training  until  the  system  was  restarted? 

g.  Explain  any  special  problems  not  listed  above. 

Did  the  temporary  delay  of  recurring  reports  adversely 
affect  3tudent  training? 
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REPORTS  GENERATION  SURVEY 


AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


4.  Do  you  consider  computer  downtime  a  major  drawback  to 
the  AIMS? 


5.  Following  computer  downtime,  was  it  necessary  to  manu¬ 
ally  record  and  input  student  data  into  the  system  to 
update  any  of  the  AIMS-generated  reports?  If  yes, 
identify  the  report  and  explain  the  problems  you  had. 


6.  Have  you  experienced  any  problems  with  these  AIMS- 

generated  reports  because  of  loss  of  data  during  down¬ 
time?  Yes _  No _ .  If  yes,  identify  the 

report  and  describe  the  problems  you  had. 


7.  Were  any  particular  problems,  not  discussed  above, 
encountered  in  up-dating  reports  following computer 
downtime? 


8.  Was  it  necessary  to  maintain  a  dual  set  of  manually 

maintained  student  records  to  insure  continuity  during 
downtime? 


9.  Please  list  and  explain  any  problems  encountered  with 
the  AIMS-generated  reports  not  previously  covered. 
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COURSE  MATERIALS  DEVELOPMENT  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


COURSE  MATERIALS  DEVELOPMENT  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


DATE 


The  purpose  of  this  survey  is  to  gather  information 
concerning  the  development  of  AIMS  instructional  materials. 
Please  answer  all  items  in  this  survey  as  factually  and 
completely  as  possible,  and  with  complete  candor.  Your 
responses  will  be  held  in  strictest  confidence.  If  you 
feel  that  you  are  not  in  a  position  to  answer  a  particular 
item  because  you  have  not  been  closely  associated  with  the 
AIMS  project,  please  circle  the  item  number  and  leave  it 
blank.  Your  comments  or  suggestions  will  be  greatly 
appreciated. 

Please  indicate  (v")  your  position  relative  to  AIMSs 


Instructor 

Instructional  Programmer 
Section  Chief 
Course  Chief 
Division  Chief 
Education  Specialist 
Training  Specialist 
Other 


COURSE  tlATERIALS  DEVELOPMENT  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 


COMMENTS 


6.  Little  difficulty  was 

experienced  when  delet¬ 
ing  or  adding  new  or  re¬ 
vised  AIMS  instructional 
materials  on-line. 


7.  A  complete  AIMS  unit  of 

instruction  can  be  changed 
overnight  eliminating  any 
delay  in  student  progress. 


8.  What  problems  were  encountered  when  inputting  and  "sav¬ 
ing"  lesson  material  during  the  process  of  entering 
courseware  on-line? 


9.  How  much  adverse  effect,  if  any,  has  computer  down-time 
had  on  entering  of  materials  on-line?  Explain. 


COURSE  MATERIALS  DEVELOPMENT  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


10. 


Did  waiting  for  system  restarts  delay  the  preparation 
of  instructional  materials?  Explain. 


11.  What  special  problems  did  you  encounter  when  logic 
coding  lesson  materials? 


Attachment  4 
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AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 
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REVISED  AND  ALTERNATE  TRAINING  MATERIALS  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 
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REVISED  AND  ALTERNATE  TRAINING  MATERIALS  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


DATE 


The  purpose  of  this  survey  is  to  gather  information 
concerning  revision  and  validation  of  AIMS  course  materials, 
as  well  as  alternate  training  material  requirements. 

Please  answer  all  items  in  this  survey  with  complete 
candor.  Your  responses  will  be  held  in  strictest  confi¬ 
dence.  The  results  of  this  survey  will  enable  the  revi¬ 
sion  and  validation  of  training  materials  to  be  more  effec¬ 
tive. 


If  you  feel  that  you  are  not  in  a  position  to  answer 
a  particular  question  because  you  have  not  been  closely 
associated  with  the  AIMS  project,  please  circle  the  item 
number  and  leave  it  blank.  Your  comments  or  suggestions 
will  be  greatly  appreciated. 

Please  indicate  ( \ your  position  relati\re  to  AIMS: 


Instructor 

Instructional  Programmer 
Section  Chief 
Course  Chief 
Division  Chief 
Education  Specialist 
Training  Specialist 
Other 


REVISED  AND  ALTERNATE  TRAINING  MATERIALS  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 


COMMENTS 


SECTION  1. 


Revision  and  Valid-| 
ation  of  Training 
Materials . 


1.  The  AIMS  provides  more 
flexibility  in  revising 
instructional  materials 
than  does  the  self-paced 
system. 


2.  Little  difficulty  has 
been  experienced  in 
revising 

a.  Individual  display 


b.  Tests 


c.  Units  of  Instruction 


d.  Flow  of  instruction 


3.  The  AIMS  review  process 
has  reduced  the  time 
normally  required  for 
introducing  new  or  re¬ 
vised  materials  into  the 
classroom. 
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REVISED  AND  ALTERNATE  TRAINING  MATERIALS  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 

YES 

NO 

COMMENTS 

SECTION  II.  Back-up  Training 
Materials 

5.  Was  it  necessary  to  pre¬ 

pare  back-up  training 
materials  to  be  used  ex¬ 
clusively  to  cover  compu¬ 
ter  downtime?  If  yes, 
please  explain. 

6.  Were  course  personnel, 

other  than  instructional 
programmers,  used  to  pre¬ 
pare  back-up  training 
materials  to  cover  com¬ 
puter  down-time?  If  yes, 
please  explain. 

7 .  Did  the  requirement  to 

develop  back-up  training 
materials  increase  the 
workload  for  the  course 
writers?  If  yes,  what 
percentage? 

8.  In  determining  course  re¬ 

source  requirements, 
should  additional  per¬ 
sonnel  be  programmed  to 
handle  the  preparation 
of  back-up  materials? 

Please  explain. 

9.  Do  you  think  there  is  a 

legitimate  requirement 
to  develop  back-up  in¬ 
structional  materials  for 
all  AIMS  instruction? 
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REVISED  AND  ALTERNATE  TRAINING  MATERIALS  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


SECTION  III.  General  Summary. 

10.  Have  you  encountered  any  special  problems  in  preparing 
the  AIMS  instructional  materials  not  noted  when  pre¬ 
paring  prior  self-paced  materials? 


11.  How  would  you  change  the  present  system  of  revising 
lesson  materials  to  insure  adequate  back-up  for  the 
AIMS  instruction? 


12.  What  problems  peculiar  to  the  AIMS  were  encountered 
when  using  small  and  large  student  groups  for  vali¬ 
dation  of  training  materials? 


Vi 


Attachment  5 


COURSE  DEVELOPMENT  INTERVIEW 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


The  Course  Development  Interview  provides  a  structure 
for  the  Interviewer  gathering  data  from  Instructional 
Developers.  Each  Developer  selected  for  an  interview  will 
be  asked  the  questions  that  appear  on  the  following  pages. 


iUlK-Mor  nuts 


COURSE  DEVELOPMENT  INTERVIEW 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


NAME: _ 

COURSE: 


%  of  Time  Spent 


a.  Planning,  Course  Outlines, 
Strategies 

b.  Original  Authoring  (writing, 
typing,  coding) 

c.  Converting  Existing  Materials 

d.  Modifying  Materials  due  to 
POI  Changes 

e.  Reviewing,  Debugging,  &  Testing, 
Materials 

f.  Revising  Materials 

g.  Coordination 
n.  Other 


f  Alt*  AUiK  not 


niMD 
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COURSE  DEVELOPMENT  SUMMARY  SHEET 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


NAME: 


POSITION: 

DATE: 


Course (s)  worked  on: 

Course 


Section 


Position 


1. 

2. 

3. 

4. 

5. 

6. 
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COURSE  DEVELOPMENT  INTERVIEW 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


NAME: _ ^ _  DATE: _ 

What  feedback  is  available  for  you  to  use  in  revising  in¬ 
structional  materials  and  tests? 


How  have  you  used  it? 


What  additional  feedback  would  you  like  to  have? 


How  would  you  use  it? 


COURSE  DEVELOPMENT  INTERVIEW 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 

NAME:  COURSE: 


What  special  qualifications  are  required  by  the  in¬ 
structional  support  staff? 


a.  Instructional  Programmers 


b.  Course  Development  Personnel 


c.  Computer  Service  Personnel 


d.  Entry  Specialists 


e.  Other  (specify) 
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COURSE  DEVELOPMENT  INTERVIEW 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


NAME: 


COURSE : 


What  administrative  and  personnel  costs  were  in¬ 
curred  to  establish  inservice  training  programs? 


COURSE  DEVELOPMENT  INTERVIEW 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


NAME: 


COURSE : 


What  difficulties  have  been  encountered  in  fitting 
the  previously  developed  self-paced  instructional  mater¬ 
ials  into  the  AIMS  management  structure? 
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COURSE  DEVELOPMENT  INTERVIEW 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


NAME: _ 

What 

entering 


COURSE : 


special  problems,  if  any,  were  encountered  when 
(inputting)  materials  on-line? 
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COURSE  DEVELOPMENT  INTERVIEW 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 

NAME: _  COURSE: _ 

What  was  the  number  and  percentage  of  students  who 
failed  to  graduate  from  each  AIMS  course? 


COURSE  DEVELOPMENT  INTERVIEW 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


NAME:  COURSE: 


What  is  the  average  training  time  for  each  AIMS  course 
How  many  graduates  were  there  in  each  course? 
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TEST  AND  EVALUATION  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 
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TEST  AND  EVALUATION  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


DATE: 


The  purpose  of  this  survey  is  to  collect  information 
on  the  tests  produced  by  the  AIMS.  The  results  will  be 
consolidated  with  other  surveys  to  determine  the  effec¬ 
tiveness  of  the  AIMS.  Your  responses  will  be  held  in 
strict  confidence. 

Please  place  a  check  (vO  mark  by  your  position  be¬ 
low. 


Instructor 

Instructional  Programmer 
Section  Chief 
Course  Chief 
Division  Chief 
Education  Specialist 
Training  Specialist 
Other 


I 
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TEST  AND  EVALUATION  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 

STRONGLY  AGREE 

AGREE 

NEUTRAL  | 

|  DISAGREE  | 

STRONGLY  DISAGREE  | 

COMMENTS 

6.  Of  the  students  who 

elect  to  take  the  pre¬ 
tests,  the  majority 
fail. 

7.  Posttests  have  been 

effective  in  measuring 
student  accomplishment. 

8.  Distractor  counts  have 

identified  weaknesses  in 
the  instructional  mater¬ 
ials. 

9 .  Pretest  and  posttest 

distractor  counts  have 
pinpointed  deficiencies 
inherent  in  the  test 
questions . 

10.  Posttests  have  proved 
to  be  highly  success¬ 
ful  in  routing  the  stu¬ 
dent  into  his/her  proper 
learning  alternative. 

1 

1 

i 


\ 

1 
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TEST  AND  EVALUATION  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


l 


T 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 


UJ 

iu 

cc 

o 

< 


UJ 

LU 

cc 

(9 

< 


< 

cc 


3 

Ui 


UJ 

UJ 

cc 

C3 

< 

(/> 

Q 


11.  Distractor  counts  have  en¬ 


abled  instructional  pro¬ 
grammers  to  make  timely 
revisions  to  questions  and 
training  materials. 


12.  Because  of  misspelling  and 
improper  phrasing,  more 
unanticipated  responses 
have  surfaced  than  ex¬ 
pected. 


13.  Unanticipated  responses 
are  useful  when  revising 
instructional  material. 


STRONGLY  DISAGREE 
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STUDENT  ATTITUDE  QUESTIONNAIRE 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


This  is  a  questionnaire  to  gather  information  about 
the  Automated  Instructional  Management  System  (AIMS) . 

There  are  no  right  or  wrong  answers.  Rather,  we  are  in¬ 
terested  in  your  candid  opinion  of  the  following  state¬ 
ments.  Your  complete  frankness  in  recording  your  opinions 
will  be  greatly  appreciated.  Individual  responses  will  be 
held  in  strictest  confidence. 


E  It" :  rcWMtoin' 


i81c®*wo  tAC m  auiK-Mor  nuco 


81 


STUDENT  ATTITUDE  QUESTIONNAIRE 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 


O  < 

Z  UJ  oc 

O  h 

OC  6C  3 

H  O  ID 

(A  <  Z 


COMMENTS 


Generally,  the  lessons 
are  hard  to  understand 


Generally,  the  lesson? 
are  too  long. 


The  level  of  reading 
skill  required  in  most 
lessons  is  too  high. 


Performance  examinations 
cover  what  is  presented 
in  the  lessons. 


Generally,  the  lessons 
seem  to  be  planned  just 
for  me. 
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STUDENT  ATTITUDE  QUESTIONNAIRE 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 

|  STRONGLY  AGREE  | 

AGREE  | 

| NEUTRAL  ! 

|  DISAGREE  | 

STRONGLY  DISAGREE  | 

COMMENTS 

11.  I  learned  the  course 
material  very  quickly 
using  this  method  of 
instruction. 

12.  I  find  myself  hurrying 
through  a  lesson  to  get 
it  over  with  rather  than 
trying  to  learn. 

13.  I  answer  questions  wrong 
intentionally  (pretest, 
posttest)  in  order  to 
get  more  instruction. 

14.  I  waste  no  time  using 

this  method  of  instruc¬ 
tion. 

15.  I  do  my  best  as  a  result 
of  this  method  of  in¬ 
struction. 
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STUDENT  ATTITUDE  QUESTIONNAIRE 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 

STRONGLY  AGREE 

AGREE  ] 

NEUTRAL 

DISAGREE 

STRONGLY  DISAGREE 

COMMENTS 

^■i 

mm 

■■■ 

■M 

16.  I  always  know  how  well  I 
am  doing  in  this  course. 

1 

1 

1 

1 

1 

17.  The  AIMS  is  a  very  effec¬ 
tive  method  of  instruc¬ 
tion  . 

1 

18.  I  feel  that  no  one  really 
cares  whether  I  learn  or 
net  using  this  method  of 
instruction. 

19.  I  feel  that  T  am  oushed 
too  rapidly  thrm  h  the 
lesson  material. 

| 

| 

| 

! 

20.  An  instructor  is  readily 
available  for  assistance. 

_i 
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STUDENT  ATTITUDE  QUESTIONNAIRE 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 

STRONGLY  AGREE 

U) 

LU 

cc 

o 

< 

— 

NEUTRAL 

DISAGREE 

STRONGLY  DISAGREE 

COMMENTS 

21.  The  instructors  can 
answer  my  questions. 

22.  There  is  a  good  working 
relationship  between  the 
instructors  and  myself. 

24.  Working  in  the  carrels 
and  other  student  posi¬ 
tions  becomes  tiresome 
over  a  long  period  of 
time. 

25.  There  are  so  many  devices 
(computer  terminals,  TV/ 
cassettes,  slide  projec¬ 
tors)  to  operate  that  it 
distracts  from  the  in¬ 
struction. 

1 
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STUDENT  ATTITUDE  QUESTIONNAIRE 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 


Z  III  s 

O  £  H 

GC  ®  3 

H  O  UJ 

CO  <  Z 


UJ  r 

g  i 

<  1 

co 


COMMENTS 


Constant  changes  from 
one  instructional  medium 
to  another  interferes 
with  learning. 


The  computer  terminals 
are  inoperative  too 
often,  which  wastes 
my  time. 


The  computer  terminal 
is  easy  to  operate. 


The  computer  terminal 
t »xt  displays  are  clear 
and  easy  to  read. 


The  amount  of  material 
presented  on  the  indivi 
dual  terminal  displays 
is  not  excessive. 


STUDENT  ATTITUDE  QUESTIONNAIRE 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 


LU 

UJ 

ce 

o 
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31.  Graphic  displays  on  the 
computer  terminal  are 
sharp  and  easy  to  under- 


COMMENTS 


stand. 


32.  I  had  no  problem  in 
learning  to  use  the 
keyboard  at  the  computer 
terminal. 


SECTION  II.  CM I  Mode  (Material 
presented  off-line 
under  computer 
management) . 

33.  I  prefer  receiving  all  my 
assignments  via  the  com¬ 
puter  terminal. 


34.  Most  of  my  time  in  the 
course  is  spent  in  the 
CMI  mode. 


35.  The  instructors  kept 

referring  me  to  computer 
terminal  for  directions 
rather  than  answering 
my  questions  about  off¬ 
line  assignments. 
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STUDENT  ATTITUDE  QUESTIONNAIRE 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 


Q 

ui  > 
a  £ 
S  o 

|  E 

Q  V) 


COMMENTS 


I  had  no  difficulty  in 
getting  to  use  a  compu¬ 
ter  terminal  whenever 
it  was  necessary. 


I  feel  that  the  AIMS 
instruction  is  too  im¬ 
personal  with  so  much 
time  spent  in  the  CMI 
mode. 


Computer  down  time  did 
not  affect  my  progress 
through  the  course. 


It  was  difficult  for  me 
to  determine  my  next 
training  task  when  the 
computer  system  was  down'. 


40.  You  always  know  exactly 
where  you  stand  in  the 
course  when  in  the  CMI 
mode. 
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STUDENT  ATTITUDE  QUESTIONNAIRE 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 


LU  H 

a  5 

O  ui 

<  z 


COMMENTS 


I  have  experienced  no 
problems  in  moving 
through  the  course  in 
CM I  mode. 


Instruction  received  via 
the  computer  terminal 
are  clear,  concise,  and 
easy  to  follow. 


I  feel  that  my  progress 
would  be  faster  if  my 
course  activities  were 
controlled  by  the  class¬ 
room  instructor. 


49.  Not  being  able  to  ask 

the  instructor  a  question 
immediately  when  a  prob¬ 
lem  occurred  has  made 
learning  more  difficult 
for  me. 


There  were  not  enough 
computer  terminals  in 
the  course  to  meet  all 
the  student  requirements. 
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STUDENT  ATTITUDE  QUESTIONNAIRE 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS,  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 


o  < 

Z  u  It 

o  £  t 

s  c  3 

h  O  U1 

(/)  <  Z 


ui  3 

8S  a 

5  O 
§  (E 

a  (A 


COMMENTS 


Too  much  waiting  occurred 
because  the  computer  ter¬ 
minals  were  down  for 
maintenance. 


Waiting  to  get  a  computer 
terminal  caused  me  to 
lose  time  in  getting 
through  a  course. 


I  would  prefer  to  have 
a  computer  terminal  at 
each  classroom  position. 


In  my  course  there  were 
too  many  students  for  the 
number  of  computer  termin¬ 
als  available. 


The  best  configuration  for 
the  AIMS  is  to  cluster  the 
computer  terminals  in  one 
classroom. 


Attachment  8 


COMPUTER-MANAGED  INSTRUCTION  ATTITUDE  QUESTIONNAIRE 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 
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COMPUTER- MANAGED  INSTRUCTION  ATTITUDE  QUESTIONNAIRE 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


The  purpose  of  the  following  questionnaire  is  to  col¬ 
lect  student  opinions  about  Computer-Managed  Instruction 
(CMI) .  There  are  no  right  or  wrong  answers.  Rather,  we 
are  interested  in  your  candid  opinion.  Individual  re¬ 
sponses  will  be  held  in  strict  confidence. 


i  wscidino  pack  blamc-mot  tiimbd 


NOTE;  The  Pretest  Questionnaire  is  designed  to  be  admin¬ 
istered  prior  to  the  CMI  course  and  the  Posttest 
Questionnaire  is  designed  to  be  adirunistered  at 
the  conclusion  of  the  course. 
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PRETEST  COMPUTER-MANAGED  INSTRUCTION  QUESTIONNAIRE 


CIRCLE  THE  LETTER  OF  YOUR  CHOICE  FOR  EACH  ITEM. 


1.  When  I  am  trying  to  learn  something,  it  is  important 
to  me  to  know  where  I  stand  in  comparison  to  others. 


(a) 

Strongly  agree 

(b) 

Agree 

(c) 

Undecided 

<d) 

Disagree 

(e) 

Strongly  disagree 

I  would 

like  to  take  a  CMI  (Computer-Managed  Ins true 

tion)  course. 

(a) 

Strongly  agree 

(b) 

Agree 

(c) 

Undecided 

(d) 

Disagree 

(e) 

Strongly  disagree 

Taking  a 

CMI  course  would  make  me 

nervous. 

(a) 

Strongly  ejree 

(b) 

Agree 

(c) 

Undecided 

(d) 

Disagree 

(e) 

Strongly  disagree 

Taking  a. 

CMI  course  would  be  more 

interesting  than 

taking  the  same  course  taught  in 

some  other  way. 

(a) 

Strongly  agree 

(b) 

Agree 

(c) 

Undecided 

(d) 

Disagree 

(e) 

Strongly  disagree 

5.  People  should  be  taught  by  other  people,  not  by 
machines. 


(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 


it 

r- 
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I  feel  very  uncomfortable  when  others  know  that  I've 
made  a  mistake. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  d  j ree 

I  think  I  would  feel  isolated  and  alone  while  taking  a 
CMI  course. 


(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

I  like  it  when  I  can  immediately  find  out  where  I  have 
made  my  mistakes. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

It  would  be  boring  to  take  a  CMI  course. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

I  think  it  would  be  easy  to  understand  the  material  in 
&  CMI  course. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

Students  are  being  treated  more  and  more  like  IBM  cards 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 


12.  Taking  a  CMI  course  would  be  too  mechanical. 


(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

13.  I  think  I  would  feel  challenged  to  do  my  best  work 
while  taking  a  CMI  course. 

(a)  All  the  time 

(b)  Most  of  the  time 

(c)  Some  of  the  time 

(d)  Only  occasionally 

(e)  Never 

14.  I  don't  like  to  have  my  errors  pointed  out  to  me. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

15.  I  would  prefer  to  have  most  courses  taught  as  CMI 
rather  than  by  other  teaching  methods. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecide  i 

(d)  Disagree 

(e)  Strongly  disagree 

16.  Most  courses  could  be  managed  more  effectively  by 
regular  teacher  than  by  computer. 

(a)  Strongly  agree- 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 


POSTTEST  COMPUTER-MANAGED  INSTRUCTION  QUESTIONNAIRE 


CIRCLE  THE  LETTER  OF  YOUR  CHOICE  FOR  EACH  ITEM.. 


1.  The  way  the  material  was  presented  to  me  made  me  feel 
that  no  one  really  cared  whether  I  learned  or  not. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

2.  I  was  not  concerned  when  I  missed  a  question  because  no 
one  was  watching  me  anyway. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

3.  The  method  by  which  I  was  told  whether  I  had  given  a 
right  or  wrong  answer  became  boring. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

4.  I  was  concerned  that  I  might  not  be  understanding  the 
material. 

(a)  All  the  time 

(b)  Most  of  the  time 

(c)  Some  of  the  time 

(d)  Only  occasionally 

(e)  Never 

5.  The  responses  to  my  answers  were  appropriate. 

(a)  All  the  time 

(b)  Most  of  the  time 

(c)  Some  of  the  time 

(d)  Only  occasionally 

(e)  Never 
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6.  I  felt  uncertain  as  to  my  performance  compared  t-  tne 
performance  of  others. 

(a)  All  the  time 

(b)  Most  of  the  time 

(c)  Some  of  the  time 

(d)  Only  occasionally 

(e)  Never 

7.  I  knew  whether  my  answers  were  correct  or  not  before 
I  was  told. 

(a)  Quite  often 

(b)  Often 

(c)  Occasionally 

(d)  Seldom 

(e)  Never 

8.  I  found  myself  just  trying  to  get  through  the  lesson 
rather  than  trying  to  learn. 

(a)  All  the  time 

(b)  Most  of  the  time 

(c)  Some  of  the  time 

(d)  Only  occasionally 

(e)  Never 

S.  I  guessed  at  the  answers  to  questions. 

(a)  Quite  often 

(b)  Often 

(c)  Occasionally 

(d)  Seldom 

(e)  Very  seldom 

10.  I  was  able  to  work  at  my  own  pace. 

(a)  All  the  time 

(b)  Most  of  the  time 

(c)  Some  of  the  time 

(d)  Only  occasionally 

(e)  Never 

11.  I  was  aware  of  efforts  to  suit  the  material  specifically 
to  me. 

(a)  Quite  often 

(b)  Often 

(c)  Occasionally 

(d)  Seldom 

(e)  Very  seldom 
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12.  I  found  it  difficult  to  concentrate  on  the  course 
material  because  of  the  machine. 

(a)  All  the  time 

(b)  Most  of  the  time 

(c)  Some  of  the  time 

(d)  Only  occasionally 

(e)  Never 

13.  Computer-managed  instruction  made  it  possible  for  me 
to  learn  quickly. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

14.  Questions  were  asked  which  were  not  relevant  to  the 
material  presented. 

(a)  All  the  time 

(b)  Most  of  the  time 

(c)  Some  of  the  time 

(d)  Only  occasionally 

(e)  Never 

15.  Based  upon  my  experience  with  this  course,  I  prefer 
CM I  to  other  methods  of  instruction. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

16.  When  I  am  trying  to  learn  something,  it  is  important 
to  me  to  know  where  I  stand  in  comparison  to  others. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

17.  I  would  like  to  take  another  CMI  course. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 
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Taking  a  CMI  course  made  me  nervous. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

The  material  managed  by  computer  was  more  interesting 
than  taking  similar  material  managed  in  some  other 
way. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

People  should  be  taught  by  other  people,  not  by 
machines . 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

I  feel  very  uncomfortable  when  others  know  that  I've 
made  a  mistake. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

I  felt  isolated  and  alone  while  working  with  the 
computer . 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

I  liked  it  when  I  was  able  to  find  out  where  I  had 
made  my  mistakes. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 
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24.  It  was  boring  to  learn  material  managed  by  computer. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 


25.  It  was  easy  to  understand  the  CMI  course  material. 


26. 


(a) 

Strongly  agree 

(b) 

Agree 

(c) 

Undecided 

(d) 

Disagree 

(e) 

Strongly  disagree 

Students 

are  being  treated  i 

(a) 

Strongly  agree 

(b) 

Agree 

(c) 

Undecided 

(d) 

Disagree 

(e) 

Strongly  disagree 

27.  Courses  managed  by  a  computer  were  too  mechanical. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

28.  I  felt  challenged  to  do  my  b&3t  work  while  in  a  CMI 
course. 


(a) 

Ail  the 

time 

(b) 

Most  of 

the  time 

(c) 

Some  of 

the  time 

(d) 

Only  occasionally 

(e) 

Never 

29.  I  don't  like  to  have  my  errors  pointed  out  to  me. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 
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30.  I  would  prefer  to  have  most  courses  taught  as  CMI 
rather  than  by  other  teaching  methods. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 

31.  Most  courses  could  be  managed  more  effectively  by  a 
regular  teacher  than  by  computer. 

(a)  Strongly  agree 

(b)  Agree 

(c)  Undecided 

(d)  Disagree 

(e)  Strongly  disagree 
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Attachment  9 


COST  CATEGORIES 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


This  list  indicates  the.  categories  of  costs  incurred 
for  Computer-Based  Training  Systems.  It  is  proposed  that 
the  Seidel  and  Wagner  (1977)  Cost-Effectiveness  Specifica¬ 
tion  be  used  for  the  collection  of  this  data  on  the  AIMS. 


COST  CATEGORIES 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


Categories 

1.  Equipment 

1.1  Computer (s) 

1.2  Terminal  (s) 

1.3  Auxiliary  AV  Devices 

1.4  Auxiliary  Memory 

1.5  Local  Interfaces 

1.6  Telephone  Lines 

1.7  Special  Lines 

1.8  Satellites 

1.9  Receivers 

1.10  Power  Generating  Equipment 

1.11  Carrels 

1.12  Other  Equipment 


2.  Facilities 

2 . 1  Classrooms 

2 . 2  Laboratories 

2 . 3  Large  Group  Instructional  Spaces 

2.4  Offices 

2 . 5  Individual  Learning  Spaces 

2 . 6  Libraries  and  Other  Information  Resource 

Centers 

2.7  Other  Facilities 


3.  Software 

3.1  Systems  Programs 

3.2  General  Applications  Programs 

3.3  Diagnostic/Test  Programs 

3.4  Utility  Programs 

3 . 5  Other  Computer  Programs 


4.  Instructional  Systems  Development* 

4.1  Analyze  (Phase  I) 

4.2  Design  Phase  (Phase  II) 

4.3  Development  Phase  (Phase  III) 

4.4  Other  Instructional  Systems  Development 

(ISD)  Activities 

*If  CMI  Subsystem  is  Operational 


COST  CATEGORIES 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


Instructional  Methods/Materials* 

5.1  Audio 

5.2  Audio/Visual 

5.3  Film  Text/Visual 

5 . 4  Lecture/Demonstration 

5.5  Group  Discussion/Seminar 

5.6  Performance/Practice 

5.7  Tutoring  (Peer  or  other) 

5.8  Printed  Test/Visual 

5.9  Other  Instructional  Methods/Materials 

5.10  Tests 

*If  CMI  Subsystem  is  Operational 


System  Management/Test 

6.1  System  Integration  Engineering 

6.2  Program  Management 

6.3  Operational  Test 

6.4  Other  Direct  Management  Costs 


Other  Direct  Costs  (not  included  in  the  above) 

7.1  Supplies 

7.2  Travel 

7.3  Consultants 

7.4  Contracts/Subcontracts 

7.5  Other  Direct  Costs 


Preparation 

8.1  Training  of  Initial  Site  Personnel  Cadre 

8.2  Site  Personnel  Pay  and  Allowances 

8.3  Other  Preparation  Costs 


Acceptance  Test/Manageirent 

9.1  Program/Project  Management 

9.2  Acceptance  Test 

9.3  Engineering  Changes 

9.4  Site  Checkout/Activation 


Equipment 

10.1  Replacement  Spares  and  Repair  Teat 
Equipment 

10.2  Other  Equipment 


System  Management 

11.1  Program/Project  Management 

11.2  Other  Direct  Management  Costs 


Other  Effectiveness  Measures 

12.1  Time  Measures 

Average  Training  Time  in  Course 
Average  Testing  Time  in  Course 
Average  Course  Time 

12.2  Achievement  Measures  (Pinal  Criterion 
Test  Results-First  Attempt) 

Accuracy  or  Speed  Scores 
Gain  Scores 

Number  of  Objectives  Passed 
%  Students  Passed 


Attachment  10 


INSTRUCTOR  TASK  LIST 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


INSTRUCTOR  TASK  LIST 


AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


This  task  list  describes  the  activities  that  an  in¬ 
structor  may  be  performing  in  courses  managed  by  the  AIMS. 
It  can  serve  as  the  basis  for  a  checklist  to  be  employed 
when  determining  the  impact  of  AIMS  upon  the  job  of  in¬ 
structor  at  USAFAS. 

I 

) 


\ 


i 


INSTRUCTOR  TASK  LIST 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


ASSIST  STUDENT:  Greet  student,  call  roll,  inform  student 
of  classroom  procedure-  update  student  records,  answer 
student's  questions. 

CONDUCT  COURSE  SUPPORT:  Check  serviceability  of  lesson 
materials,  and  equipment.  Store,  organize  and  inventory 
lesson  materials,  tools,  and  equipment.  Secure  classroom, 
review  and  update  lesson  material.  Maintain  technical 
orders,  maintain  classroom  expendable  supplies. 

OVERSEE  MEASUREMENT  CENTER:  Provide  examination  material, 
update  student  record,  grade/critique  test,  assign  student, 
and  monitor  test  center. 

CONDUCT  STUDENT  SUPPORT:  Maintain  student  records,  sched¬ 
ule/monitor  individual  assistance  and  student  breaks.  Give 
safety  briefing,  enforce  safety,  insure  proper  classroom 
management,  lighting  and  temperature,  conduct  preassessment 
briefing  and  orientation.  Monitor  student  progress. 

CONDUCT  OTHER  SUPPORT:  Maintain  unclassified  file  and  pub¬ 
lication  file.  Review  incoming  distribution,  inspect 
facility,  receive  and  assist  visiting  official,  attend 
meetings. 

CONDUCT  GRADUATION:  Conduct  final  critique,  figure  time 
in  course  and  grade  average,  and  conduct  ceremony. 

CONDUCT  DEMONSTRATION/PERFORMANCE:  Conduct  demonstration/ 
performance  to  give  student  hands-on  training  on  equipment 
related  to  written  material. 

COUNSEL/CRITIQUE  STUDENT:  Counsel  student  on  progression 
and  improvement  areas.  Counsel  and  assist  students  with 
morale,  welfare  and  disciplinary  problems.  Take  necessary 
corrective  action  required  to  maintain  discipline. 

REVIEW  COURSE  MATERIAL:  Review  new  and  old  course  and 
test  material. 

GRADE  TESTS:  Review  and  grade  tests,  inform  student  of 
grade  and  area  needing  review. 

UPDATE  LESSON  MATERIAL:  Update  lesson  material  to  reflect 
changes  in  controlling  documents. 

DEVELOP  NEW  COURSE  MATERIAL:  Develop  new  course  and  test 
material. 


INSTRUCTOR  TASK  LIST 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


OPERATE  AIMS  TERMINALS:  Request  records  and  reports,  ob¬ 
tain  computer  prescription,  override  computer,  request  stu 
dent  data,  locate  student,  update  student  record,  update 
resource  availability. 

REQUEST  MAINTENANCE  ASSISTANCE:  Contact  major  source  for 
repair.  Maintains  log  of  when  assistance  was  called  for 
and  when  corrective  action  was  taken.  Maintains  log  of 
down  time. 

CONSTANT  MANNING  REQUIREMENT:  Monitor  classroom. 

OTHER: 


From:  Dallman,  et  al.,  1979. 
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Attachment  11 


INSTRUCTOR  ATTITUDE  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


INSTRUCTOR  ATTITUDE  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


The  purpose  of  this  survey  is  to  determine  your 
activities  and  opinions  regarding  the  various  aspects  of 
the  AIMS  and  Computer-Managed  Instruction  (CMI) .  Please 
answer  all  items  in  the  survey  with  complete  candor.  Your 
individual  responses  will  be  held  in  strictest  confidence. 


PART  I :  Background 

1.  Course: 

2.  Date: _ 

3.  How  long  have  you  been  an  instructor  in  this  course 

(months)  ? _ _ _ 

4.  Previous  instructor  position (s)  was  in  a:  (circle 
appropriate  choice (s) ) 

a.  Conventional  classroom  environment. 

b.  Self-paced  multimedia  instructional  environment. 

c.  Computer  managed  self-paced  instructional  environ¬ 
ment. 

d.  No  previous  instructor  experience. 

e.  Others  (specify) _ 


STRONGLY  DISAGREE 


PART  II:  Attitudes  (Cont) 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


CHECK  APPROPRIATE  BLOCK  TO  INDICATE  YOUR 
ATTITUDE  TOWARDS  THE  AIMS.  EXPLAIN  YOUR 
SELECTION  IN  THE  COMMENTS  SECTION  AS 
NECESSARY. 

STRONGLY  AGREE 

AGREE 

NEUTRAL 

DISAGREE 

STRONGLY  DISAGREE 

COMMENTS 

6.  It  takes  an  experienced  in¬ 

structor  to  manage  this  type 
of  instruction. 

7.  The  students'  attitudes  to¬ 

ward  the  course  are  very 
favorable. 

8.  The  students  use  too  much 

time  learning  to  operate  in¬ 
structional  equipment  (other 
than  computer  terminals) . 

9.  Students  appear  to  be  very 

interested  in  the  course 
content. 

10.  The  students  are  very  en¬ 
thusiastic  about  the 
course. 

PART  II:  Attitudes  (Cont) 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


CHECK  APPROPRIATE  BLOCK  TO  INDICATE  YOUR 
ATTITUDE  TOWARDS  THE  AIMS.  EXPLAIN  YOUR 
SELECTION  IN  THE  COMMENTS  SECTION  AS 
NECESSARY. 
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COMMENTS 


The  AIMS  teaches  as  well 
as  other  methods,  and  in 
less  time. 


The  instructional  proce¬ 
dures  in  this  course  need 
a  major  revision. 


Instructional  materials 
must  be  more  thoroughly 
organized  then  for  group- 
paced  teaching  methods. 


The  instructional  materials 
concentrate  on  specific 
skills  and  knowledges  and 
excludes  "nice  to  know" 
lesson  material. 


The  instructional  materials 
and  media  are  suitable  for 
the  type  of  student  input 
into  this  course. 


PART  II:  Attitudes  (Cont) 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


CHECK  APPROPRIATE  BLOCK  TO  INDICATE  YOUR 
ATTITUDE  TOWARDS  THE  AIMS.  EXPLAIN  YOUR 
SELECTION  IN  THE  COMMENTS  SECTION  AS 
NECESSARY. 
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COMMENTS 

21.  Using  this  method  the  in¬ 
structor  to  student  ratio 
can  be  reduced  over  group- 
paced  methods. 

22.  Students  experienced  little 
difficulty  in  operating  the 
computer  terminal. 

23.  Computer-managed  instruc¬ 
tion  will  enable  a  rapid 
increase  in  student  inputs 
should  need  arise. 

24.  Looking  to  the  future,  I 

feel  that  CMI  will  replace 
other  methods  of  instruc¬ 
tion. 

25.  Computer-managed  instruc¬ 
tion  is  a  potential  threat 
to  the  jobs  of  the  instruc¬ 
tors. 
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PART  II:  Attitudes  (Cont) 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


CHECK  APPROPRIATE  BLOCK  TO  INDICATE  YOUR 
ATTITUDE  TOWARDS  THE  AIMS.  EXPLAIN  YOUR 
SELECTION  IN  THE  COMMENTS  SECTION  AS 
NECESSARY. 

STRONGLY  AGREE 

AGREE 

NEUTRAL 

DISAGREE 

STRONGLY  DISAGREE 

COMMENTS 

mm 

mm 

26.  Any  capable  instructor  can 

prepare  lesson  materials  for 
CMI  courses. 

1 

1 

1 

l 

I 

27.  All  instructors  should  take 
a  CMI  course  prior  to  teach¬ 
ing  in  this  mode. 

1 

1 

1 

I 

1 

I 

I 

I 

1 

1 

l 
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PART  II:  Attitudes  (Cont) 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIM5!) 


28.  Do  you  feel  that  the  AIMS  has  done  a 

good  job  of  managing  the  students  off¬ 
line  activities.  Please  explain. 


29.  Has  there  been  an  opposition  to  AIMS 

among  the  instructors?  If  yes,  please 
explain. 


30.  Do  you  think  that  the  CMI  aspect  of 

the  AIMS  has  accelerated  the  students' 
progress  through  the  course? 
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PART  III:  Activities  (Cont) 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


HOW  MUCH  Of  YOUR  TIME  DURING 
INSTRUCTIONAL  DU  FS  FOR  THIS  COURSE 
DID  YOU  SPEND  ON  EACH  OF  THE 
FOLLOWING  ACTIVITIES,’  CHECK  THE 
MOST  APPROPRIATE  RESPONSE  AND 
EXPLAIN  RESPONSE  IN  COMMENTS 
AS  NECESSARY. 


*  * 


t-  •-* 

z  o 

J5  i 

2  5 


COMMENTS 


31.  Monitoring  students 


32.  Individually  counseling 
students  on  lesson 
material. 


33.  Counseling  students  in 
small  groups  on  lesson 
material. 


34.  Administering  tests. 


Giving  direction  on 
performance  exercises, 


Maintaining  student 
records. 


’■W".  t- 


PART  Ills  Activities  (Cont) 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


HOW  MUCH  OF  YOUR  TIME  DURING 
INSTRUCTIONAL  DUTIES  FOR  THIS  COURSE 
DID  YOU  SPEND  ON  EACH  OF  THE 

FOLLOWING  ACTIVITIES?  CHECK  THE 

MOST  APPROPRIATE  RESPONSE  AND 

EXPLAIN  RESPONSE  IN  COMMENTS 

AS  NECESSARY. 

MORE  THAN  HALF 

SUBSTANTIAL  AMOUNT 

FAIR  AMOUNT 

VERY  LITTLE 

NONE 

COMMENTS 

43.  Counseling  students 
on  matters  other  than 
lesson  material.  (Career 
guidance) 

44.  Recording  instructor 

comments  about  students. 

1 

1 

1 

1 

\ 

45.  Faculty  board  actions. 

i 

* 

-tx 

46.  Interaction  with  other 
instructors . 

$ 

$ 

i 

£ 

47.  Administrative  duties 
not  mentioned.  (List 
under  comments  if  they 
tzke  much  of  your  time.) 

k 

48.  Preparing  for  classroom 
instruction. 

i 

.* 

s 
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PART  Ills  Activities  (Cont) 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


HOW  MUCH  OF  YOUR  TIME  DURING 
INSTRUCTIONAL  DUTIES  FOR  THIS  COURSE 
DID  YOU  SPEND  ON  EACH  OF  THE 
FOLLOWING  ACTIVITIES?  CHECK  THE 
MOST  APPROPRIATE  RESPONSE  AND 
EXPLAIN  RESPONSE  IN  COMMENTS 
AS  NECESSARY. 
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COMMENTS 


130 


'i"v~ 


PART  III:  Activities  (Cont) 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS' 

51.  Which  of  the  activities  do  you  feel  take  too  much  of 
your  instructor  time?  Indicate  by  number  from  the  pre¬ 
ceding  list  (Items  31-50) . 


52.  Which  of  the  activities  do  you  feel  you  need  to  de¬ 
vote  more  of  your  instructor  time  to?  Indicate  by  number 
from  the  preceding  list  (Items  31-50) . 


PART  IV:  Suggestions/Observations 


53.  Of  the  classroom  environments  you  are  familiar  with, 
which  of  the  following  would  you  prefer  to  be  an  instructor 
in?  (Circle  best  response) 

a.  Conventional  classroom  environment. 

b.  Self-paced  classroom  environment. 

c.  Computer  managed  self-paced  classroom  environ¬ 
ment. 

d.  Others  (specify) _ _. 


54.  What  training  (if  any)  do  you  think  is  needed  for 
instructors  using  the  AIMS  instructional  method? 
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PART  IV:  Suggestions/Observations  (Cont) 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 

55.  What  guidelines  or  advice  would  you  give  a  new  in 
structor  who  is  going  to  use  this  teaching  method  for 
the  first  time? 


56.  Which  of  the  following  items  would  you  identify  as 
advantages  of  CMI?  (Rank  in  order  of  importance.) 

_ a.  Reduced  training  time. 

_ b.  Higher  student  achievement. 

_ c.  Better  student  motivation. 

_ d.  Lower  failure  rate. 

_ e.  Preferred  by  most  students. 

_ f.  Fewer  instructors  required. 

_ g.  Greater  staff  and  faculty  interest. 

_ h.  Better  performance  after  training. 

_ i.  Promotes  development  of  the  learning  process. 

_ j.  An  effective  teacher-aide  to  the  classroom 

instructor. 

_ k.  Other  (specify) 

57.  What  are  the  disadvantages  of  this  method  ol  in¬ 
struction? 


,  v  *.-r  i  ui  '»■  \t*n  . .  |i|^|WVVfrJ~lJ‘l  ^ 


PART  IV:  Suggescions/Observations  (Cont) 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


58.  What  problems  have  you  encountered  using  this  method 
of  instruction? 


59.  If  you  were  the  Director  of  Training,  what  would  you 
do  to  improve  the  AIMS? 


60.  What  is  your  frank  opinion  abort  the  relative  worth 
of  computer-managed  instruction? 


61.  How  has  the  implementation  of  AIMS  assisted  you  in 
monitoring  student  progress  in  the  course?  Please  explain. 


62.  Do  you  feel  that  using  the  computer  to  monitor  stu¬ 
dent  progress  has,  in  fact,  been  more  effective  than  the 
method  used  previously?  Please  explain. 


63.  Can  you  define  or  describe  any  unique  or  unusual  prob¬ 
lems  encountered  in  preparing  instructional  materials  not 
identified  when  preparing  self-paced  material? 


64.  What  is  your  frank  opinion  of  using  the  computer  to 
randomly  generate  test  items? 


65.  Has  the  identification,  recording,  and  hard  copy 
summation  of  test  performance  been  helpful  in  pinpointing 
weak  areas  of  instruction? 


PART  IV:  Suggestions/Observations  (Cont) 

AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 

66.  What  problems,  if  any,  have  you  experienced  with 
student  queueing  (either  waiting  to  be  tested  or  to 
start  the  next  task)? 


67.  Has  the  AIMS  enabled  the  student  to  achieve  his/her 
objectives  in  less  time  than  the  prior  self-paced  course 
format?  Yes  No 


68.  Do  you  find  that  the  AIMS  student  graduation  predic¬ 
tions  have  been  more  timely  and  functional  than  those  re¬ 
ceived  under  the  previous  self-paced  operation?  Please 
explain. 
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Attachment  12 

STAFF  AND  FACULTY  ATTITUDE  SURVEY 
/  tTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


STAFF  AND  FACULTY  ATTITUDE  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 

DATE 


This  is  a  survey  to  gather  information  relative  to 
the  Automated  Instructional  Management  System  (AIMS) . 

Please  answer  all  items  in  this  survey  with  complete 
candor.  Your  responses  will  be  held  in  strictest  con¬ 
fidence.  If  you  feel  that  you  are  not  in  a  position  to 
answer  a  particular  item  because  you  have  not  been  closely 
associated  with  the  AIMS  project,  circle  the  item  number 
and  leave  it  blank.  Your  comments  or  suggestions  will  be 
greatly  appreciated. 

Please  place  a  check  (l^)  mark  by  your  position  below. 


Instructor 

Education  Specialist 
Training  Specialist 
Section  Chief 
Course  Chief 
Division  Chief 


STAFF  AND  FACULTY  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
IMOICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 


O  < 

z  tu  oc 

O  "  H 

oc  CE  3 

h  o  HI 

0)  <  Z 


COMMENTS 


The  AIMS  teaches  as  well 
as  other  methods,  and  in 
less  time. 


The  AIMS  concentrates  on 
specific  skills  and  know¬ 
ledges  and  excludes  "nice 
to  know"  items. 


AIMS  students  are  more 
apt  to  retain  skills  and 
knowledges  learned  for  a 
longer  period  after  com¬ 
pletion. 


Under  the  AIMS  program 
students  have  varied 
repetition  or  practice 


The  AIMS  instructional 
strategies  are  suitable 
for  all  types  of  stu¬ 
dents  found  in  the 


courses. 
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STAFF  AND  FACULTY  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 


The  AIMS  instructional 
materials  are  designed 
to  fit  all  levels  of 
students. 


The  AIMS  courses  do  an 
outstanding  job  of  teach¬ 
ing  students  to  meet 
performance  objectives. 


The  AIMS  enables  better 
students  to  proceed  at 
an  accelerated  rate  and 
they  invariably  complete 
the  course  in  less  than 
programmed  time. 


Under  the  AIMS,  course 
lengths  can  be  adjusted 
with  minimum  delay  based 
on  student  progress. 
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COMMENTS 


The  AIMS  enables  the 
instructor  to  organize 
course  materials  more 
effectively  to  meet  in¬ 
dividual  students  needs. 
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STAFF  AND  FACULTY  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 
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COMMENTS 


16.  Slow  learners,  who  would 
be  academic  failures  in 
group-paced  courses,  are 
able  to  attain  the  mini¬ 
mum  performance  standards 
under  the  AIMS. 


17.  The  AIMS  provides  more 

opportunity  for  students 
to  react  to  simulated 
on-the-job  performance 
standards  under  the  AIMS. 


18.  The  AIMS  provides  a 

learning  environment  that 
matches  the  students 
learning  performance  with 
materials  appropriate  to 
his  learning  rate. 


19.  The  greatest  value  of 

the  AIMS  is  that  it  en¬ 
ables  the  student  to 
learn  at  his/her  own 
pace. 


20.  The  most  valid  criterion 
for  judging  the  success 
of  the  AIMS  is  the  stu¬ 
dents'  performance  after 
training. 
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STAFF  AND  FACULTY  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


PLEASE  CHECK  THE  APPROPRIATE  BLOCK  TO 
INDICATE  YOUR  OPINION  OF  THE  FOLLOWING 
STATEMENTS.  COMMENTS  EXPLAINING  YOUR 
SELECTION  WILL  BE  APPRECIATED. 
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COMMENTS 


21.  Over  a  period  of  time, 
the  novelty  effect  of 
the  AIMS  wears  off  and 
the  students  tend  to 
become  bored. 


22.  It  is  not  anticipated 

that  the  implementation 
of  the  AIMS  will  have 
any  impact  on  the  man¬ 
power  structure  of  the 
courses  involved. 


23.  The  administrative  re¬ 
quirements  (maintaining 
student  records  and  re¬ 
ports)  for  the  AIMS 
courses  do  not  differ 
greatly  from  other  self- 
paced  courses. 


24.  On-line  pretests/post¬ 
tests  have  proved  to  be 
highly  effective  as  a 
teaching/learning  stra¬ 
tegy  . 


25.  The  AIMS  has  significantly 
reduced  the  instructor  to 
student  ratio  resulting 
in  a  saving  of  instructor 
personnel. 
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STAFF  AND  FACULTY  SURVEY 
AUTOMATED  INSTRUCTIONAL  MANAGEMENT  SYSTEM  (AIMS) 


28.  The  time  required  for  development  of  AIMS  instructional 
materials  - 

a.  exceeds  that  required  for  group-paced  instruction 

by  (indicated  percentage)  _ %. 

b.  is  about  the  same  as  that  required  for  group- 

paced  instruction  _ . 

c.  is  less  than  that  required  for  group-paced  in¬ 
struction  by  (indicate  percentage)  _ %. 


SECTION  II.  Open-Ended  Questions. 

29.  In  your  opinion,  what  are  the  advantages  and  disad¬ 
vantages  of  the  AIMS  instruction  when  compared  to  other 
(group-paced/self-paced)  instructional  methods? 


Advantages 

Disadvantages 

30.  If  you  were  in  a  position  to  dictate  policy  for  future 
self-paced  courses  in  other  service  schools,  what  initial 
guidelines  would  you  give  your  project  officer? 


31.  What  is  your  frank  opinion  concerning  the  relative 
worth  of  self-paced  versus  group-paced  instruction? 
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i  us  coast  guard  academy  / 

1  USA  IRANSPORTATION  SCHOOL  DIRECTORATE  OF  TRAINING  ♦  DOCTRINE 
1  USA  INFANTRY  SCHOOL  LIBRARY  / 

1  USA  mP  ♦  CHEM  Sch/TNG  CEN  ♦  FT.  MCCLEO.AN  ATTn:  ATZN-PTS 

1  USA  mP  ♦  CHEM  ScH/TNG  CEN  ♦  FT.  MCCLEU-an  DIR.  COMBAT  DEVELOPMENT 

1  USA  mP  ♦  CHEM  Sch/TNG  CtN  ♦  FT.  MCCLELLAN  DIR.  TRAINING  DEVELOPMENT 
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1  USA  mP  ♦  CHEM  ScH/TNU  CEN  ♦  FT.  MCCLELLAN  ATTN:  ATZN-MP-ACt 
1  USA  INSTITUTE  Of  aUMINISTHaTIUn  ATTN:  RESIDENT  TRAINING  MANAGEMENT 

i  uSa  i-ield  artillery  school  mor.us  swett  library 

1  USA  INSTITUTE  OF  aOMINI STHATIOn  ACADEMIC  LIBRARY 
1  USa  cNGINEER  SCHOOL  L l BHAH Y  ANU  LEARNING  RESOURCES  CENTER 
1  USA  aRMOH  SCHOOL  (USARhS)  ATTM  LIBRARY 
1  US  CoAST  GUARD  ACADEMY  LIBRARY 

1  US  AwMY  INTELLIGENCE  CENftR  ♦  SCHOOL  ATTN*.  ATSI-TD 

i  us  army  intelligence  center  ♦  school  attn:  atsi-rm-m 
1  DEPARTMENT  OF  The  AIR  FORCt  atR  university  library  (ATC) 

1  HQ  TrADOC  training  OEVELOF'MENT  institute 

?  BRITISH  EMBASSY  3KIUSH  UtFENCt  STAFF 
?  CANADIAN  JOINT  STAFF 
1  COLS  ( M )  LIBRARY 
1  FRENCH  ARMY  ATTACHE 

i  Austrian  Embassy  defense,  military  and  air  attache 

3  CANADIAN  DEFENCE  lIaISON  STAFF  ATTNj  COUNSELLOR,  DEFENCE  R  AND  D 
1  RO>Al  NETHERLANDS  EMBASSY  MILITARY  ATTACHE 
?  CANADIAN  FORCES  PERSONNEL  APPL  RSCH  UNIT 
1  ARMY  PERSONNEL  RESEARCH  ESTABLISHMENT 
6  LIBRaRY  OF  CONGRESS  EXCHANGE  aNL'  GIFT  DIV 
1  DEFENSE  TECHNICAL  INFORMATION  cEN  ATTN;  DTlC-DDA-2 
HO  LIBRARY  OF  CONGRESS  UNIT  DOCUMENTS  EXPEDITING  PRqJECT 

i  us  Government  printing  oFl.  library,  public  documents  department 
i  us  Government  printing  ofc  lihhary  and  statutory,  lib  div  isll) 

1  THE  army  LIBRAHy  ATTn:  ARmY  STUDIES  StC 

3  royal  army  educational  cohps  CENiRt  army  school  of  Training  support 


